Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 08:11:06.767287 20910 test_util.cc:268] Using random seed: 1123512196
I20240503 08:11:06.779860 20910 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:06.779955 20910 ts_itest-base.cc:113] --------------
I20240503 08:11:06.779997 20910 ts_itest-base.cc:114] 6 tablet servers
I20240503 08:11:06.780076 20910 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:11:06.780141 20910 ts_itest-base.cc:116] --------------
2024-05-03T08:11:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:06Z Disabled control of system clock
I20240503 08:11:06.805733 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.107.190:38313
--webserver_interface=127.20.107.190
--webserver_port=0
--builtin_ntp_servers=127.20.107.148:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.107.190:38313
--location_mapping_cmd=/tmp/dist-test-taskZB6CEZ/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 08:11:07.150076 20922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:07.150406 20922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:07.150503 20922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:07.158464 20922 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:07.158605 20922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:11:07.158653 20922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:07.158717 20922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:11:07.158767 20922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:11:07.168465 20922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskZB6CEZ/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.20.107.190:38313
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.107.190:38313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.107.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:07.169814 20922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:07.171567 20922 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:07.183629 20927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.184195 20930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.185089 20928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.185881 20922 server_base.cc:1034] running on GCE node
I20240503 08:11:07.186797 20922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:07.188347 20922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:07.189556 20922 hybrid_clock.cc:648] HybridClock initialized: now 1714723867189498 us; error 61 us; skew 500 ppm
I20240503 08:11:07.189916 20922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:07.192309 20922 webserver.cc:460] Webserver started at http://127.20.107.190:39715/ using document root <none> and password file <none>
I20240503 08:11:07.193126 20922 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:07.193245 20922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:07.193545 20922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:07.196833 20922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4bedc607804b4c579e99e7e8aea3f5b2"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-4tl1"
I20240503 08:11:07.197641 20922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4bedc607804b4c579e99e7e8aea3f5b2"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-4tl1"
I20240503 08:11:07.204332 20922 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20240503 08:11:07.207902 20936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.209510 20922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:07.209707 20922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4bedc607804b4c579e99e7e8aea3f5b2"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-4tl1"
I20240503 08:11:07.209924 20922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:07.235422 20922 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:07.236366 20922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:07.236651 20922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:07.256781 20922 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.190:38313
I20240503 08:11:07.256798 20987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.190:38313 every 8 connection(s)
I20240503 08:11:07.258313 20922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:11:07.262689 20988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:07.268570 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 20922
I20240503 08:11:07.268985 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:11:07.272313 20988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2: Bootstrap starting.
I20240503 08:11:07.276502 20988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:07.277912 20988 log.cc:824] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:07.280493 20988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2: No bootstrap required, opened a new log
I20240503 08:11:07.288153 20988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bedc607804b4c579e99e7e8aea3f5b2" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 38313 } }
I20240503 08:11:07.288552 20988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:07.288664 20988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bedc607804b4c579e99e7e8aea3f5b2, State: Initialized, Role: FOLLOWER
I20240503 08:11:07.289359 20988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bedc607804b4c579e99e7e8aea3f5b2" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 38313 } }
I20240503 08:11:07.289621 20988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:07.289757 20988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:07.289901 20988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:07.292157 20988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bedc607804b4c579e99e7e8aea3f5b2" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 38313 } }
I20240503 08:11:07.292629 20988 leader_election.cc:302] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bedc607804b4c579e99e7e8aea3f5b2; no voters:
I20240503 08:11:07.293324 20988 leader_election.cc:288] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:07.293457 20993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:07.294731 20993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4bedc607804b4c579e99e7e8aea3f5b2, State: Running, Role: LEADER
I20240503 08:11:07.295293 20993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bedc607804b4c579e99e7e8aea3f5b2" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 38313 } }
I20240503 08:11:07.295665 20988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:07.298202 20995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bedc607804b4c579e99e7e8aea3f5b2. Latest consensus state: current_term: 1 leader_uuid: "4bedc607804b4c579e99e7e8aea3f5b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bedc607804b4c579e99e7e8aea3f5b2" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 38313 } } }
I20240503 08:11:07.298614 20995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:07.299223 20994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bedc607804b4c579e99e7e8aea3f5b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bedc607804b4c579e99e7e8aea3f5b2" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 38313 } } }
I20240503 08:11:07.299520 20994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:07.302194 21000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:07.309440 21000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:07.320191 21000 catalog_manager.cc:1347] Generated new cluster ID: d9fd797eb17940cabadf0455a56a21a1
I20240503 08:11:07.320319 21000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:07.331279 21000 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:07.332952 21000 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:07.345281 21000 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4bedc607804b4c579e99e7e8aea3f5b2: Generated new TSK 0
I20240503 08:11:07.346200 21000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:07.357290 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.129:0
--local_ip_for_outbound_sockets=127.20.107.129
--webserver_interface=127.20.107.129
--webserver_port=0
--tserver_master_addrs=127.20.107.190:38313
--builtin_ntp_servers=127.20.107.148:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:11:07.732091 21012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:07.732393 21012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:07.732512 21012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:07.740367 21012 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:07.740495 21012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:07.740669 21012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.129
I20240503 08:11:07.750151 21012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.107.129
--webserver_port=0
--tserver_master_addrs=127.20.107.190:38313
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.129
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:07.751500 21012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:07.753302 21012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:07.766108 21017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.766885 21020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.767587 21018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.768440 21012 server_base.cc:1034] running on GCE node
I20240503 08:11:07.769196 21012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:07.770212 21012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:07.771451 21012 hybrid_clock.cc:648] HybridClock initialized: now 1714723867771391 us; error 51 us; skew 500 ppm
I20240503 08:11:07.771811 21012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:07.774291 21012 webserver.cc:460] Webserver started at http://127.20.107.129:38929/ using document root <none> and password file <none>
I20240503 08:11:07.775110 21012 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:07.775239 21012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:07.775581 21012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:07.778364 21012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "18d9776f5d634ec2b167dc225659e4fd"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-4tl1"
I20240503 08:11:07.779270 21012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "18d9776f5d634ec2b167dc225659e4fd"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-4tl1"
I20240503 08:11:07.784797 21012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 08:11:07.788205 21026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.789572 21012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:07.789747 21012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "18d9776f5d634ec2b167dc225659e4fd"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-4tl1"
I20240503 08:11:07.789948 21012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:07.813745 21012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:07.814623 21012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:07.814918 21012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:07.815860 21012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:07.817370 21012 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:07.817476 21012 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.817593 21012 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:07.817677 21012 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.853184 21012 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.129:39921
I20240503 08:11:07.853237 21138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.129:39921 every 8 connection(s)
I20240503 08:11:07.854485 21012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:07.858134 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 21012
I20240503 08:11:07.858345 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:11:07.861601 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.130:0
--local_ip_for_outbound_sockets=127.20.107.130
--webserver_interface=127.20.107.130
--webserver_port=0
--tserver_master_addrs=127.20.107.190:38313
--builtin_ntp_servers=127.20.107.148:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:07.869019 21139 heartbeater.cc:347] Connected to a master server at 127.20.107.190:38313
I20240503 08:11:07.869333 21139 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.870162 21139 heartbeater.cc:511] Master 127.20.107.190:38313 requested a full tablet report, sending...
I20240503 08:11:07.931552 20953 ts_manager.cc:194] Registered new tserver with Master: 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:07.933398 20953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.129:52593
W20240503 08:11:08.222249 21143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:08.222573 21143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:08.223163 21143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:08.230840 21143 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:08.231010 21143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:08.231185 21143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.130
I20240503 08:11:08.240725 21143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.107.130
--webserver_port=0
--tserver_master_addrs=127.20.107.190:38313
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.130
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:08.242113 21143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:08.243805 21143 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:08.256139 21149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.256596 21150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.257019 21152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:08.257899 21143 server_base.cc:1034] running on GCE node
I20240503 08:11:08.258536 21143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:08.260076 21143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:08.261397 21143 hybrid_clock.cc:648] HybridClock initialized: now 1714723868261323 us; error 72 us; skew 500 ppm
I20240503 08:11:08.261750 21143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:08.264416 21143 webserver.cc:460] Webserver started at http://127.20.107.130:32997/ using document root <none> and password file <none>
I20240503 08:11:08.265332 21143 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:08.265465 21143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:08.265787 21143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:08.268472 21143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3b6eea641d214c79988ec6ad8821a4bf"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-4tl1"
I20240503 08:11:08.269493 21143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3b6eea641d214c79988ec6ad8821a4bf"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-4tl1"
I20240503 08:11:08.275945 21143 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.006s
I20240503 08:11:08.280028 21158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.281483 21143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:11:08.281641 21143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3b6eea641d214c79988ec6ad8821a4bf"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-4tl1"
I20240503 08:11:08.281849 21143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:08.294464 21143 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:08.295331 21143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:08.295607 21143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:08.296685 21143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:08.298368 21143 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:08.298468 21143 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.298584 21143 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:08.298658 21143 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.334806 21270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.130:35525 every 8 connection(s)
I20240503 08:11:08.334811 21143 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.130:35525
I20240503 08:11:08.336251 21143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:11:08.340735 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 21143
I20240503 08:11:08.340956 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:11:08.344604 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.131:0
--local_ip_for_outbound_sockets=127.20.107.131
--webserver_interface=127.20.107.131
--webserver_port=0
--tserver_master_addrs=127.20.107.190:38313
--builtin_ntp_servers=127.20.107.148:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:08.350719 21271 heartbeater.cc:347] Connected to a master server at 127.20.107.190:38313
I20240503 08:11:08.351069 21271 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:08.352126 21271 heartbeater.cc:511] Master 127.20.107.190:38313 requested a full tablet report, sending...
I20240503 08:11:08.397666 20953 ts_manager.cc:194] Registered new tserver with Master: 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525)
I20240503 08:11:08.398594 20953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.130:46481
W20240503 08:11:08.686683 21274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:08.687004 21274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:08.687126 21274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:08.694478 21274 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:08.694597 21274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:08.694747 21274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.131
I20240503 08:11:08.704044 21274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.107.131
--webserver_port=0
--tserver_master_addrs=127.20.107.190:38313
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.131
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:08.705406 21274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:08.707008 21274 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:08.719380 21280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.719833 21281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.720372 21283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:08.721109 21274 server_base.cc:1034] running on GCE node
I20240503 08:11:08.721680 21274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:08.722591 21274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:08.723783 21274 hybrid_clock.cc:648] HybridClock initialized: now 1714723868723732 us; error 47 us; skew 500 ppm
I20240503 08:11:08.724083 21274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:08.726301 21274 webserver.cc:460] Webserver started at http://127.20.107.131:40303/ using document root <none> and password file <none>
I20240503 08:11:08.727033 21274 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:08.727128 21274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:08.727443 21274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:08.730177 21274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-4tl1"
I20240503 08:11:08.730895 21274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-4tl1"
I20240503 08:11:08.735919 21274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:11:08.739178 21289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.740455 21274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 08:11:08.740608 21274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-4tl1"
I20240503 08:11:08.740805 21274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:08.777076 21274 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:08.778424 21274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:08.778697 21274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:08.779803 21274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:08.781424 21274 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:08.781536 21274 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.781656 21274 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:08.781734 21274 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.816972 21274 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.131:37537
I20240503 08:11:08.817072 21401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.131:37537 every 8 connection(s)
I20240503 08:11:08.818313 21274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:08.824123 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 21274
I20240503 08:11:08.824343 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:11:08.828151 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.132:0
--local_ip_for_outbound_sockets=127.20.107.132
--webserver_interface=127.20.107.132
--webserver_port=0
--tserver_master_addrs=127.20.107.190:38313
--builtin_ntp_servers=127.20.107.148:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:08.836179 21402 heartbeater.cc:347] Connected to a master server at 127.20.107.190:38313
I20240503 08:11:08.836539 21402 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:08.837631 21402 heartbeater.cc:511] Master 127.20.107.190:38313 requested a full tablet report, sending...
I20240503 08:11:08.887559 20953 ts_manager.cc:194] Registered new tserver with Master: 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537)
I20240503 08:11:08.889046 20953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.131:38767
I20240503 08:11:08.936358 21139 heartbeater.cc:503] Master 127.20.107.190:38313 was elected leader, sending a full tablet report...
W20240503 08:11:09.188167 21405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:09.188463 21405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:09.188609 21405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:09.196672 21405 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:09.196805 21405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:09.196998 21405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.132
I20240503 08:11:09.206146 21405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.107.132
--webserver_port=0
--tserver_master_addrs=127.20.107.190:38313
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.132
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:09.207477 21405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:09.209301 21405 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:09.222476 21411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:09.222905 21414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:09.223613 21412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:09.224207 21405 server_base.cc:1034] running on GCE node
I20240503 08:11:09.224826 21405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:09.225843 21405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:09.227059 21405 hybrid_clock.cc:648] HybridClock initialized: now 1714723869227011 us; error 46 us; skew 500 ppm
I20240503 08:11:09.227396 21405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:09.229805 21405 webserver.cc:460] Webserver started at http://127.20.107.132:46757/ using document root <none> and password file <none>
I20240503 08:11:09.231031 21405 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:09.231153 21405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:09.231477 21405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:09.234148 21405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3e04594b1a5344d9bfefa4de05d70842"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-4tl1"
I20240503 08:11:09.234886 21405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3e04594b1a5344d9bfefa4de05d70842"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-4tl1"
I20240503 08:11:09.240123 21405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240503 08:11:09.243633 21420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:09.245169 21405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 08:11:09.245335 21405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3e04594b1a5344d9bfefa4de05d70842"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-4tl1"
I20240503 08:11:09.245528 21405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:09.269418 21405 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:09.270390 21405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:09.270687 21405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:09.271987 21405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:09.274489 21405 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:09.274646 21405 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:09.274807 21405 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:09.274883 21405 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:09.312141 21405 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.132:41603
I20240503 08:11:09.312192 21532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.132:41603 every 8 connection(s)
I20240503 08:11:09.313509 21405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:09.317665 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 21405
I20240503 08:11:09.317836 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:11:09.321481 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.133:0
--local_ip_for_outbound_sockets=127.20.107.133
--webserver_interface=127.20.107.133
--webserver_port=0
--tserver_master_addrs=127.20.107.190:38313
--builtin_ntp_servers=127.20.107.148:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:09.326740 21533 heartbeater.cc:347] Connected to a master server at 127.20.107.190:38313
I20240503 08:11:09.327080 21533 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:09.328091 21533 heartbeater.cc:511] Master 127.20.107.190:38313 requested a full tablet report, sending...
I20240503 08:11:09.373073 20953 ts_manager.cc:194] Registered new tserver with Master: 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603)
I20240503 08:11:09.374382 20953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.132:41219
I20240503 08:11:09.401612 21271 heartbeater.cc:503] Master 127.20.107.190:38313 was elected leader, sending a full tablet report...
W20240503 08:11:09.674381 21536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:09.674748 21536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:09.674885 21536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:09.682516 21536 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:09.682698 21536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:09.682863 21536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.133
I20240503 08:11:09.692574 21536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.107.133
--webserver_port=0
--tserver_master_addrs=127.20.107.190:38313
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.133
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:09.693967 21536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:09.695789 21536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:09.708088 21543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:09.708645 21542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:09.709646 21545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:09.710232 21536 server_base.cc:1034] running on GCE node
I20240503 08:11:09.710850 21536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:09.712283 21536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:09.713482 21536 hybrid_clock.cc:648] HybridClock initialized: now 1714723869713435 us; error 44 us; skew 500 ppm
I20240503 08:11:09.713806 21536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:09.716212 21536 webserver.cc:460] Webserver started at http://127.20.107.133:40743/ using document root <none> and password file <none>
I20240503 08:11:09.717006 21536 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:09.717128 21536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:09.717433 21536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:09.720062 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-4tl1"
I20240503 08:11:09.720788 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-4tl1"
I20240503 08:11:09.725778 21536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20240503 08:11:09.729074 21551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:09.730397 21536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:11:09.730552 21536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-4tl1"
I20240503 08:11:09.730758 21536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:09.760553 21536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:09.761476 21536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:09.761726 21536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:09.762657 21536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:09.764225 21536 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:09.764319 21536 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:09.764405 21536 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:09.764459 21536 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:09.802918 21536 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.133:43423
I20240503 08:11:09.803009 21663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.133:43423 every 8 connection(s)
I20240503 08:11:09.804301 21536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:09.811777 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 21536
I20240503 08:11:09.811995 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:11:09.816084 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.134:0
--local_ip_for_outbound_sockets=127.20.107.134
--webserver_interface=127.20.107.134
--webserver_port=0
--tserver_master_addrs=127.20.107.190:38313
--builtin_ntp_servers=127.20.107.148:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:09.821197 21664 heartbeater.cc:347] Connected to a master server at 127.20.107.190:38313
I20240503 08:11:09.821964 21664 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:09.822778 21664 heartbeater.cc:511] Master 127.20.107.190:38313 requested a full tablet report, sending...
I20240503 08:11:09.869402 20953 ts_manager.cc:194] Registered new tserver with Master: 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:09.870674 20953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.133:55371
I20240503 08:11:09.891846 21402 heartbeater.cc:503] Master 127.20.107.190:38313 was elected leader, sending a full tablet report...
W20240503 08:11:10.166937 21667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:10.167244 21667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:10.167353 21667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:10.175479 21667 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:10.175627 21667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:10.175839 21667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.134
I20240503 08:11:10.185854 21667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.107.134
--webserver_port=0
--tserver_master_addrs=127.20.107.190:38313
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.134
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:10.187237 21667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:10.189137 21667 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:10.202217 21673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:10.202879 21676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:10.203241 21674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:10.203976 21667 server_base.cc:1034] running on GCE node
I20240503 08:11:10.204841 21667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:10.205852 21667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:10.207047 21667 hybrid_clock.cc:648] HybridClock initialized: now 1714723870206996 us; error 53 us; skew 500 ppm
I20240503 08:11:10.207360 21667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:10.209630 21667 webserver.cc:460] Webserver started at http://127.20.107.134:36657/ using document root <none> and password file <none>
I20240503 08:11:10.210362 21667 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:10.210456 21667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:10.210762 21667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:10.213534 21667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6503743925434401a74aae303628e415"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-4tl1"
I20240503 08:11:10.214313 21667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6503743925434401a74aae303628e415"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-4tl1"
I20240503 08:11:10.219492 21667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:11:10.222770 21682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:10.224061 21667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:10.224222 21667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6503743925434401a74aae303628e415"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-4tl1"
I20240503 08:11:10.225021 21667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:10.251237 21667 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:10.252111 21667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:10.252386 21667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:10.253885 21667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:10.255518 21667 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:10.255618 21667 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:10.255744 21667 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:10.255834 21667 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:10.293951 21667 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.134:42791
I20240503 08:11:10.294005 21794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.134:42791 every 8 connection(s)
I20240503 08:11:10.295293 21667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:11:10.297597 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 21667
I20240503 08:11:10.297763 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:11:10.308959 21795 heartbeater.cc:347] Connected to a master server at 127.20.107.190:38313
I20240503 08:11:10.309269 21795 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:10.310066 21795 heartbeater.cc:511] Master 127.20.107.190:38313 requested a full tablet report, sending...
I20240503 08:11:10.357390 20953 ts_manager.cc:194] Registered new tserver with Master: 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:10.358343 20910 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 08:11:10.358426 20953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.134:60469
I20240503 08:11:10.377436 21533 heartbeater.cc:503] Master 127.20.107.190:38313 was elected leader, sending a full tablet report...
I20240503 08:11:10.380613 20910 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 08:11:10.873533 21664 heartbeater.cc:503] Master 127.20.107.190:38313 was elected leader, sending a full tablet report...
I20240503 08:11:11.361809 21795 heartbeater.cc:503] Master 127.20.107.190:38313 was elected leader, sending a full tablet report...
I20240503 08:11:14.134512 20953 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41046:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:14.198647 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc3222fc2a54a2087b347efd3b18f0a (DEFAULT_TABLE table=rebalance_test_table_0 [id=af682d14b61c432aa57e42ec67d0e792]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:14.200336 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc3222fc2a54a2087b347efd3b18f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:14.198558 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9dc83cb659489a82b0fff6694cb1a7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=af682d14b61c432aa57e42ec67d0e792]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:14.202605 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9dc83cb659489a82b0fff6694cb1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:14.215109 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 0e049e82d55547288f5911a11a47ffb6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=af682d14b61c432aa57e42ec67d0e792]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:14.216655 21730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e049e82d55547288f5911a11a47ffb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:14.216287 21598 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9dc83cb659489a82b0fff6694cb1a7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=af682d14b61c432aa57e42ec67d0e792]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:14.217571 21598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9dc83cb659489a82b0fff6694cb1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:14.225740 21468 tablet_service.cc:1467] Processing CreateTablet for tablet 0e049e82d55547288f5911a11a47ffb6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=af682d14b61c432aa57e42ec67d0e792]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:14.225740 21074 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc3222fc2a54a2087b347efd3b18f0a (DEFAULT_TABLE table=rebalance_test_table_0 [id=af682d14b61c432aa57e42ec67d0e792]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:14.227316 21074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc3222fc2a54a2087b347efd3b18f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:14.228060 21468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e049e82d55547288f5911a11a47ffb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:14.229537 21073 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9dc83cb659489a82b0fff6694cb1a7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=af682d14b61c432aa57e42ec67d0e792]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:14.229972 21073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9dc83cb659489a82b0fff6694cb1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:14.215672 21599 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc3222fc2a54a2087b347efd3b18f0a (DEFAULT_TABLE table=rebalance_test_table_0 [id=af682d14b61c432aa57e42ec67d0e792]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:14.233552 21599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc3222fc2a54a2087b347efd3b18f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:14.238750 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 0e049e82d55547288f5911a11a47ffb6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=af682d14b61c432aa57e42ec67d0e792]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:14.240207 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e049e82d55547288f5911a11a47ffb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:14.249756 21820 tablet_bootstrap.cc:492] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:14.264485 21820 tablet_bootstrap.cc:654] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:14.268285 21822 tablet_bootstrap.cc:492] T 9fc3222fc2a54a2087b347efd3b18f0a P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:14.269073 21819 tablet_bootstrap.cc:492] T 9fc3222fc2a54a2087b347efd3b18f0a P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:14.269325 21820 log.cc:824] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:14.273455 21825 tablet_bootstrap.cc:492] T 0e049e82d55547288f5911a11a47ffb6 P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:14.273758 21822 tablet_bootstrap.cc:654] T 9fc3222fc2a54a2087b347efd3b18f0a P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:14.273758 21819 tablet_bootstrap.cc:654] T 9fc3222fc2a54a2087b347efd3b18f0a P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:14.275249 21819 log.cc:824] T 9fc3222fc2a54a2087b347efd3b18f0a P 74c8d441f0f6425b88ec78c8b0d7f3d5: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:14.275696 21822 log.cc:824] T 9fc3222fc2a54a2087b347efd3b18f0a P 17af6d2a2fd14b6d93e232da6aa8f6a9: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:14.276841 21823 tablet_bootstrap.cc:492] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:14.281661 21825 tablet_bootstrap.cc:654] T 0e049e82d55547288f5911a11a47ffb6 P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:14.282270 21822 tablet_bootstrap.cc:492] T 9fc3222fc2a54a2087b347efd3b18f0a P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:14.282622 21822 ts_tablet_manager.cc:1367] T 9fc3222fc2a54a2087b347efd3b18f0a P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20240503 08:11:14.283143 21821 tablet_bootstrap.cc:492] T 0e049e82d55547288f5911a11a47ffb6 P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:14.284195 21820 tablet_bootstrap.cc:492] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:14.284581 21820 ts_tablet_manager.cc:1367] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.004s
I20240503 08:11:14.290930 21821 tablet_bootstrap.cc:654] T 0e049e82d55547288f5911a11a47ffb6 P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:14.291131 21819 tablet_bootstrap.cc:492] T 9fc3222fc2a54a2087b347efd3b18f0a P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:14.291416 21819 ts_tablet_manager.cc:1367] T 9fc3222fc2a54a2087b347efd3b18f0a P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.003s
I20240503 08:11:14.292340 21821 log.cc:824] T 0e049e82d55547288f5911a11a47ffb6 P 3e04594b1a5344d9bfefa4de05d70842: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:14.295166 21823 tablet_bootstrap.cc:654] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:14.295014 21820 raft_consensus.cc:348] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.295441 21820 raft_consensus.cc:374] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:14.295545 21820 raft_consensus.cc:729] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:14.296205 21820 consensus_queue.cc:260] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.296602 21825 log.cc:824] T 0e049e82d55547288f5911a11a47ffb6 P 3b6eea641d214c79988ec6ad8821a4bf: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:14.297206 21823 log.cc:824] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:14.298813 21822 raft_consensus.cc:348] T 9fc3222fc2a54a2087b347efd3b18f0a P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.299284 21822 raft_consensus.cc:374] T 9fc3222fc2a54a2087b347efd3b18f0a P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:14.299391 21822 raft_consensus.cc:729] T 9fc3222fc2a54a2087b347efd3b18f0a P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:14.304626 21822 consensus_queue.cc:260] T 9fc3222fc2a54a2087b347efd3b18f0a P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.308408 21819 raft_consensus.cc:348] T 9fc3222fc2a54a2087b347efd3b18f0a P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.308938 21819 raft_consensus.cc:374] T 9fc3222fc2a54a2087b347efd3b18f0a P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:14.309057 21819 raft_consensus.cc:729] T 9fc3222fc2a54a2087b347efd3b18f0a P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:14.309820 21819 consensus_queue.cc:260] T 9fc3222fc2a54a2087b347efd3b18f0a P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.331712 21819 ts_tablet_manager.cc:1398] T 9fc3222fc2a54a2087b347efd3b18f0a P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.040s user 0.013s sys 0.009s
I20240503 08:11:14.332280 21819 tablet_bootstrap.cc:492] T 1d9dc83cb659489a82b0fff6694cb1a7 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:14.334245 21822 ts_tablet_manager.cc:1398] T 9fc3222fc2a54a2087b347efd3b18f0a P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.051s user 0.023s sys 0.001s
I20240503 08:11:14.334677 21821 tablet_bootstrap.cc:492] T 0e049e82d55547288f5911a11a47ffb6 P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:14.334707 21822 tablet_bootstrap.cc:492] T 1d9dc83cb659489a82b0fff6694cb1a7 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:14.334982 21821 ts_tablet_manager.cc:1367] T 0e049e82d55547288f5911a11a47ffb6 P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.006s
I20240503 08:11:14.336050 21825 tablet_bootstrap.cc:492] T 0e049e82d55547288f5911a11a47ffb6 P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:14.336397 21825 ts_tablet_manager.cc:1367] T 0e049e82d55547288f5911a11a47ffb6 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.063s user 0.005s sys 0.017s
I20240503 08:11:14.337282 21822 tablet_bootstrap.cc:654] T 1d9dc83cb659489a82b0fff6694cb1a7 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:14.339071 21819 tablet_bootstrap.cc:654] T 1d9dc83cb659489a82b0fff6694cb1a7 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:14.342361 21823 tablet_bootstrap.cc:492] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:14.342721 21823 ts_tablet_manager.cc:1367] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.066s user 0.011s sys 0.012s
I20240503 08:11:14.346021 21820 ts_tablet_manager.cc:1398] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.061s user 0.018s sys 0.028s
I20240503 08:11:14.346563 21820 tablet_bootstrap.cc:492] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:14.347134 21819 tablet_bootstrap.cc:492] T 1d9dc83cb659489a82b0fff6694cb1a7 P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:14.347033 21821 raft_consensus.cc:348] T 0e049e82d55547288f5911a11a47ffb6 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:14.347645 21821 raft_consensus.cc:374] T 0e049e82d55547288f5911a11a47ffb6 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:14.347754 21819 ts_tablet_manager.cc:1367] T 1d9dc83cb659489a82b0fff6694cb1a7 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.004s
I20240503 08:11:14.347796 21821 raft_consensus.cc:729] T 0e049e82d55547288f5911a11a47ffb6 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:14.347793 21825 raft_consensus.cc:348] T 0e049e82d55547288f5911a11a47ffb6 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:14.348196 21825 raft_consensus.cc:374] T 0e049e82d55547288f5911a11a47ffb6 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:14.348310 21820 tablet_bootstrap.cc:654] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:14.348306 21825 raft_consensus.cc:729] T 0e049e82d55547288f5911a11a47ffb6 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:14.348788 21819 raft_consensus.cc:348] T 1d9dc83cb659489a82b0fff6694cb1a7 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.349025 21819 raft_consensus.cc:374] T 1d9dc83cb659489a82b0fff6694cb1a7 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:14.349124 21819 raft_consensus.cc:729] T 1d9dc83cb659489a82b0fff6694cb1a7 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:14.348835 21821 consensus_queue.cc:260] T 0e049e82d55547288f5911a11a47ffb6 P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:14.351496 21822 tablet_bootstrap.cc:492] T 1d9dc83cb659489a82b0fff6694cb1a7 P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:14.351434 21825 consensus_queue.cc:260] T 0e049e82d55547288f5911a11a47ffb6 P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:14.351698 21822 ts_tablet_manager.cc:1367] T 1d9dc83cb659489a82b0fff6694cb1a7 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.004s
I20240503 08:11:14.352933 21822 raft_consensus.cc:348] T 1d9dc83cb659489a82b0fff6694cb1a7 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.353195 21822 raft_consensus.cc:374] T 1d9dc83cb659489a82b0fff6694cb1a7 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:14.353294 21822 raft_consensus.cc:729] T 1d9dc83cb659489a82b0fff6694cb1a7 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:14.353593 21822 consensus_queue.cc:260] T 1d9dc83cb659489a82b0fff6694cb1a7 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.354202 21822 ts_tablet_manager.cc:1398] T 1d9dc83cb659489a82b0fff6694cb1a7 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:14.354566 21819 consensus_queue.cc:260] T 1d9dc83cb659489a82b0fff6694cb1a7 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.355265 21819 ts_tablet_manager.cc:1398] T 1d9dc83cb659489a82b0fff6694cb1a7 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:11:14.356843 21821 ts_tablet_manager.cc:1398] T 0e049e82d55547288f5911a11a47ffb6 P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.022s user 0.017s sys 0.002s
I20240503 08:11:14.357115 21825 ts_tablet_manager.cc:1398] T 0e049e82d55547288f5911a11a47ffb6 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.021s user 0.011s sys 0.000s
I20240503 08:11:14.358434 21820 tablet_bootstrap.cc:492] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:14.358599 21820 ts_tablet_manager.cc:1367] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20240503 08:11:14.359586 21820 raft_consensus.cc:348] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.359791 21820 raft_consensus.cc:374] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:14.359874 21820 raft_consensus.cc:729] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:14.360097 21820 consensus_queue.cc:260] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.360586 21820 ts_tablet_manager.cc:1398] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:11:14.360414 21823 raft_consensus.cc:348] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:14.360836 21823 raft_consensus.cc:374] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:14.360989 21823 raft_consensus.cc:729] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
W20240503 08:11:14.361042 21140 tablet.cc:2369] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:14.361697 21823 consensus_queue.cc:260] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:14.363572 21823 ts_tablet_manager.cc:1398] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.021s user 0.012s sys 0.002s
I20240503 08:11:14.470932 21829 raft_consensus.cc:482] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:14.471195 21829 raft_consensus.cc:504] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.472976 21829 leader_election.cc:288] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:14.480681 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9dc83cb659489a82b0fff6694cb1a7" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" is_pre_election: true
I20240503 08:11:14.481309 21357 raft_consensus.cc:2454] T 1d9dc83cb659489a82b0fff6694cb1a7 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 0.
I20240503 08:11:14.481177 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9dc83cb659489a82b0fff6694cb1a7" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" is_pre_election: true
I20240503 08:11:14.481673 21619 raft_consensus.cc:2454] T 1d9dc83cb659489a82b0fff6694cb1a7 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 0.
I20240503 08:11:14.482050 21027 leader_election.cc:302] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d9776f5d634ec2b167dc225659e4fd, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:14.482493 21829 raft_consensus.cc:2781] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:14.482635 21829 raft_consensus.cc:482] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:14.482730 21829 raft_consensus.cc:3037] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:14.484719 21829 raft_consensus.cc:504] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.485368 21829 leader_election.cc:288] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 election: Requested vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:14.485903 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9dc83cb659489a82b0fff6694cb1a7" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
I20240503 08:11:14.485914 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9dc83cb659489a82b0fff6694cb1a7" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5"
I20240503 08:11:14.486142 21357 raft_consensus.cc:3037] T 1d9dc83cb659489a82b0fff6694cb1a7 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:14.486117 21619 raft_consensus.cc:3037] T 1d9dc83cb659489a82b0fff6694cb1a7 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:14.489583 21619 raft_consensus.cc:2454] T 1d9dc83cb659489a82b0fff6694cb1a7 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 1.
I20240503 08:11:14.489622 21357 raft_consensus.cc:2454] T 1d9dc83cb659489a82b0fff6694cb1a7 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 1.
I20240503 08:11:14.490053 21027 leader_election.cc:302] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d9776f5d634ec2b167dc225659e4fd, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:14.490365 21829 raft_consensus.cc:2781] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:14.490824 21829 raft_consensus.cc:686] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [term 1 LEADER]: Becoming Leader. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Running, Role: LEADER
I20240503 08:11:14.491380 21829 consensus_queue.cc:237] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.495906 20951 catalog_manager.cc:5484] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129). New cstate: current_term: 1 leader_uuid: "18d9776f5d634ec2b167dc225659e4fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:14.498236 21839 raft_consensus.cc:482] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:14.498456 21839 raft_consensus.cc:504] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:14.500289 21839 leader_election.cc:288] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603), 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525)
I20240503 08:11:14.507134 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e049e82d55547288f5911a11a47ffb6" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842" is_pre_election: true
I20240503 08:11:14.507668 21488 raft_consensus.cc:2454] T 0e049e82d55547288f5911a11a47ffb6 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 0.
I20240503 08:11:14.507917 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e049e82d55547288f5911a11a47ffb6" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf" is_pre_election: true
I20240503 08:11:14.508239 21684 leader_election.cc:302] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e04594b1a5344d9bfefa4de05d70842, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:14.508395 21226 raft_consensus.cc:2454] T 0e049e82d55547288f5911a11a47ffb6 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 0.
I20240503 08:11:14.508569 21839 raft_consensus.cc:2781] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:14.508678 21839 raft_consensus.cc:482] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:14.508777 21839 raft_consensus.cc:3037] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:14.510766 21839 raft_consensus.cc:504] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:14.511346 21839 leader_election.cc:288] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 election: Requested vote from peers 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603), 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525)
I20240503 08:11:14.511754 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e049e82d55547288f5911a11a47ffb6" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842"
I20240503 08:11:14.511920 21488 raft_consensus.cc:3037] T 0e049e82d55547288f5911a11a47ffb6 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:14.511888 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e049e82d55547288f5911a11a47ffb6" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf"
I20240503 08:11:14.512076 21226 raft_consensus.cc:3037] T 0e049e82d55547288f5911a11a47ffb6 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:14.514240 21226 raft_consensus.cc:2454] T 0e049e82d55547288f5911a11a47ffb6 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 1.
I20240503 08:11:14.514513 21488 raft_consensus.cc:2454] T 0e049e82d55547288f5911a11a47ffb6 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 1.
I20240503 08:11:14.514662 21685 leader_election.cc:302] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:14.514832 21829 raft_consensus.cc:482] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:14.514977 21839 raft_consensus.cc:2781] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:14.515092 21829 raft_consensus.cc:504] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.515451 21839 raft_consensus.cc:686] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [term 1 LEADER]: Becoming Leader. State: Replica: 6503743925434401a74aae303628e415, State: Running, Role: LEADER
I20240503 08:11:14.515910 21829 leader_election.cc:288] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423), 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537)
I20240503 08:11:14.515952 21839 consensus_queue.cc:237] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:14.516523 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc3222fc2a54a2087b347efd3b18f0a" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" is_pre_election: true
I20240503 08:11:14.516551 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc3222fc2a54a2087b347efd3b18f0a" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" is_pre_election: true
I20240503 08:11:14.516727 21619 raft_consensus.cc:2454] T 9fc3222fc2a54a2087b347efd3b18f0a P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 0.
I20240503 08:11:14.516775 21357 raft_consensus.cc:2454] T 9fc3222fc2a54a2087b347efd3b18f0a P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 0.
I20240503 08:11:14.517200 21029 leader_election.cc:302] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 18d9776f5d634ec2b167dc225659e4fd, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:14.517778 21829 raft_consensus.cc:2781] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:14.517869 21829 raft_consensus.cc:482] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:14.517977 21829 raft_consensus.cc:3037] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:14.520490 21829 raft_consensus.cc:504] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.521092 21829 leader_election.cc:288] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 election: Requested vote from peers 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423), 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537)
I20240503 08:11:14.520984 20951 catalog_manager.cc:5484] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6503743925434401a74aae303628e415 (127.20.107.134). New cstate: current_term: 1 leader_uuid: "6503743925434401a74aae303628e415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:14.521416 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc3222fc2a54a2087b347efd3b18f0a" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
I20240503 08:11:14.521654 21619 raft_consensus.cc:3037] T 9fc3222fc2a54a2087b347efd3b18f0a P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:14.521574 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc3222fc2a54a2087b347efd3b18f0a" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5"
I20240503 08:11:14.521807 21357 raft_consensus.cc:3037] T 9fc3222fc2a54a2087b347efd3b18f0a P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:14.524389 21619 raft_consensus.cc:2454] T 9fc3222fc2a54a2087b347efd3b18f0a P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 1.
I20240503 08:11:14.524528 21357 raft_consensus.cc:2454] T 9fc3222fc2a54a2087b347efd3b18f0a P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 1.
I20240503 08:11:14.524827 21029 leader_election.cc:302] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 18d9776f5d634ec2b167dc225659e4fd; no voters:
I20240503 08:11:14.525110 21829 raft_consensus.cc:2781] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:14.525270 21829 raft_consensus.cc:686] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [term 1 LEADER]: Becoming Leader. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Running, Role: LEADER
I20240503 08:11:14.525542 21829 consensus_queue.cc:237] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:14.527878 20951 catalog_manager.cc:5484] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129). New cstate: current_term: 1 leader_uuid: "18d9776f5d634ec2b167dc225659e4fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: HEALTHY } } }
W20240503 08:11:14.549738 21796 tablet.cc:2369] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:14.559507 21665 tablet.cc:2369] T 1d9dc83cb659489a82b0fff6694cb1a7 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:14.568782 21534 tablet.cc:2369] T 0e049e82d55547288f5911a11a47ffb6 P 3e04594b1a5344d9bfefa4de05d70842: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:14.573797 21403 tablet.cc:2369] T 1d9dc83cb659489a82b0fff6694cb1a7 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:14.592680 21272 tablet.cc:2369] T 0e049e82d55547288f5911a11a47ffb6 P 3b6eea641d214c79988ec6ad8821a4bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:14.898950 21849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:14.899252 21849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:14.906389 21849 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:11:14.924401 21839 consensus_queue.cc:1035] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:14.932806 21839 consensus_queue.cc:1035] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 08:11:15.016125 21829 consensus_queue.cc:1035] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:15.052697 21829 consensus_queue.cc:1035] T 1d9dc83cb659489a82b0fff6694cb1a7 P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:15.104585 21882 consensus_queue.cc:1035] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:15.149474 21619 raft_consensus.cc:1261] T 9fc3222fc2a54a2087b347efd3b18f0a P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Refusing update from remote peer 18d9776f5d634ec2b167dc225659e4fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:11:15.150184 21900 consensus_queue.cc:1035] T 9fc3222fc2a54a2087b347efd3b18f0a P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 282.668 ms
time per row: 0.141334 ms
I20240503 08:11:15.255582 21884 mvcc.cc:205] Tried to move back new op lower bound from 7023508992475496448 to 7023508990058471424. Current Snapshot: MvccSnapshot[applied={T|T < 7023508992448516096}]
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:11:16.326898 20952 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41046:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:16.336231 21468 tablet_service.cc:1467] Processing CreateTablet for tablet 3818719d4dc64f60b25efdc49c7e602b (DEFAULT_TABLE table=rebalance_test_table_1 [id=9cbec6ff03fa44e1ac768c1717ba0aca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:16.336719 21468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3818719d4dc64f60b25efdc49c7e602b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:16.337028 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 3818719d4dc64f60b25efdc49c7e602b (DEFAULT_TABLE table=rebalance_test_table_1 [id=9cbec6ff03fa44e1ac768c1717ba0aca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:16.337433 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3818719d4dc64f60b25efdc49c7e602b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:16.338011 21074 tablet_service.cc:1467] Processing CreateTablet for tablet d192acaf3cf74b508944391d49c95cbc (DEFAULT_TABLE table=rebalance_test_table_1 [id=9cbec6ff03fa44e1ac768c1717ba0aca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:16.338394 21074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d192acaf3cf74b508944391d49c95cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:16.341296 21205 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3dfcb5d6c943bea49ebfb7ebb50796 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9cbec6ff03fa44e1ac768c1717ba0aca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:16.341681 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3dfcb5d6c943bea49ebfb7ebb50796. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:16.343572 21337 tablet_service.cc:1467] Processing CreateTablet for tablet d192acaf3cf74b508944391d49c95cbc (DEFAULT_TABLE table=rebalance_test_table_1 [id=9cbec6ff03fa44e1ac768c1717ba0aca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:16.343974 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d192acaf3cf74b508944391d49c95cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:16.345261 21598 tablet_service.cc:1467] Processing CreateTablet for tablet d192acaf3cf74b508944391d49c95cbc (DEFAULT_TABLE table=rebalance_test_table_1 [id=9cbec6ff03fa44e1ac768c1717ba0aca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:16.345777 21598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d192acaf3cf74b508944391d49c95cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:16.349589 21467 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3dfcb5d6c943bea49ebfb7ebb50796 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9cbec6ff03fa44e1ac768c1717ba0aca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:16.350150 21467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3dfcb5d6c943bea49ebfb7ebb50796. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:16.351212 21729 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3dfcb5d6c943bea49ebfb7ebb50796 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9cbec6ff03fa44e1ac768c1717ba0aca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:16.351264 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 3818719d4dc64f60b25efdc49c7e602b (DEFAULT_TABLE table=rebalance_test_table_1 [id=9cbec6ff03fa44e1ac768c1717ba0aca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:16.351416 21926 tablet_bootstrap.cc:492] T 3818719d4dc64f60b25efdc49c7e602b P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:16.351728 21729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3dfcb5d6c943bea49ebfb7ebb50796. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:16.352042 21730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3818719d4dc64f60b25efdc49c7e602b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:16.353207 21926 tablet_bootstrap.cc:654] T 3818719d4dc64f60b25efdc49c7e602b P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:16.362221 21927 tablet_bootstrap.cc:492] T d192acaf3cf74b508944391d49c95cbc P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:16.363169 21930 tablet_bootstrap.cc:492] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:16.364431 21929 tablet_bootstrap.cc:492] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:16.365118 21930 tablet_bootstrap.cc:654] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:16.365373 21927 tablet_bootstrap.cc:654] T d192acaf3cf74b508944391d49c95cbc P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:16.366950 21929 tablet_bootstrap.cc:654] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:16.371516 21930 tablet_bootstrap.cc:492] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:16.372208 21930 ts_tablet_manager.cc:1367] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240503 08:11:16.373443 21930 raft_consensus.cc:348] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:16.373935 21929 tablet_bootstrap.cc:492] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:16.374176 21929 ts_tablet_manager.cc:1367] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 08:11:16.375058 21930 raft_consensus.cc:374] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:16.375258 21929 raft_consensus.cc:348] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:16.375443 21930 raft_consensus.cc:729] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:16.375592 21929 raft_consensus.cc:374] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:16.375716 21929 raft_consensus.cc:729] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:16.376228 21929 consensus_queue.cc:260] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:16.377301 21929 ts_tablet_manager.cc:1398] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:16.377564 21929 tablet_bootstrap.cc:492] T 3818719d4dc64f60b25efdc49c7e602b P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:16.376212 21930 consensus_queue.cc:260] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:16.379452 21929 tablet_bootstrap.cc:654] T 3818719d4dc64f60b25efdc49c7e602b P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:16.380916 21934 tablet_bootstrap.cc:492] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:16.382862 21934 tablet_bootstrap.cc:654] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:16.383225 21937 tablet_bootstrap.cc:492] T d192acaf3cf74b508944391d49c95cbc P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:16.387248 21937 tablet_bootstrap.cc:654] T d192acaf3cf74b508944391d49c95cbc P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:16.389329 21934 tablet_bootstrap.cc:492] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:16.389750 21926 tablet_bootstrap.cc:492] T 3818719d4dc64f60b25efdc49c7e602b P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:16.390067 21926 ts_tablet_manager.cc:1367] T 3818719d4dc64f60b25efdc49c7e602b P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.039s user 0.017s sys 0.000s
I20240503 08:11:16.391072 21934 ts_tablet_manager.cc:1367] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20240503 08:11:16.391520 21926 raft_consensus.cc:348] T 3818719d4dc64f60b25efdc49c7e602b P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:16.391942 21926 raft_consensus.cc:374] T 3818719d4dc64f60b25efdc49c7e602b P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:16.392115 21926 raft_consensus.cc:729] T 3818719d4dc64f60b25efdc49c7e602b P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:16.392473 21934 raft_consensus.cc:348] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:16.392546 21926 consensus_queue.cc:260] T 3818719d4dc64f60b25efdc49c7e602b P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:16.393522 21930 ts_tablet_manager.cc:1398] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.021s user 0.004s sys 0.004s
I20240503 08:11:16.393568 21929 tablet_bootstrap.cc:492] T 3818719d4dc64f60b25efdc49c7e602b P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:16.393779 21929 ts_tablet_manager.cc:1367] T 3818719d4dc64f60b25efdc49c7e602b P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240503 08:11:16.394146 21927 tablet_bootstrap.cc:492] T d192acaf3cf74b508944391d49c95cbc P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:16.394341 21927 ts_tablet_manager.cc:1367] T d192acaf3cf74b508944391d49c95cbc P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.019s
I20240503 08:11:16.394572 21937 tablet_bootstrap.cc:492] T d192acaf3cf74b508944391d49c95cbc P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:16.394584 21929 raft_consensus.cc:348] T 3818719d4dc64f60b25efdc49c7e602b P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:16.394802 21929 raft_consensus.cc:374] T 3818719d4dc64f60b25efdc49c7e602b P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:16.394872 21929 raft_consensus.cc:729] T 3818719d4dc64f60b25efdc49c7e602b P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:16.395045 21937 ts_tablet_manager.cc:1367] T d192acaf3cf74b508944391d49c95cbc P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 08:11:16.395140 21929 consensus_queue.cc:260] T 3818719d4dc64f60b25efdc49c7e602b P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:16.395133 21927 raft_consensus.cc:348] T d192acaf3cf74b508944391d49c95cbc P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:16.395408 21927 raft_consensus.cc:374] T d192acaf3cf74b508944391d49c95cbc P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:16.395561 21927 raft_consensus.cc:729] T d192acaf3cf74b508944391d49c95cbc P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:16.395777 21929 ts_tablet_manager.cc:1398] T 3818719d4dc64f60b25efdc49c7e602b P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:16.395872 21927 consensus_queue.cc:260] T d192acaf3cf74b508944391d49c95cbc P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:16.396098 21934 raft_consensus.cc:374] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:16.396270 21934 raft_consensus.cc:729] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:16.396602 21934 consensus_queue.cc:260] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:16.396757 21926 ts_tablet_manager.cc:1398] T 3818719d4dc64f60b25efdc49c7e602b P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:11:16.397332 21927 ts_tablet_manager.cc:1398] T d192acaf3cf74b508944391d49c95cbc P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:11:16.398288 21926 tablet_bootstrap.cc:492] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:16.400002 21926 tablet_bootstrap.cc:654] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:16.402771 21937 raft_consensus.cc:348] T d192acaf3cf74b508944391d49c95cbc P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:16.403160 21937 raft_consensus.cc:374] T d192acaf3cf74b508944391d49c95cbc P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:16.403326 21937 raft_consensus.cc:729] T d192acaf3cf74b508944391d49c95cbc P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:16.403715 21937 consensus_queue.cc:260] T d192acaf3cf74b508944391d49c95cbc P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:16.404249 21934 ts_tablet_manager.cc:1398] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.013s user 0.005s sys 0.003s
I20240503 08:11:16.406080 21934 tablet_bootstrap.cc:492] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:16.406081 21937 ts_tablet_manager.cc:1398] T d192acaf3cf74b508944391d49c95cbc P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.011s user 0.002s sys 0.002s
I20240503 08:11:16.408448 21934 tablet_bootstrap.cc:654] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:16.409886 21926 tablet_bootstrap.cc:492] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:16.410138 21926 ts_tablet_manager.cc:1367] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20240503 08:11:16.411012 21926 raft_consensus.cc:348] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:16.411192 21926 raft_consensus.cc:374] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:16.411299 21926 raft_consensus.cc:729] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:16.411397 21934 tablet_bootstrap.cc:492] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:16.411569 21934 ts_tablet_manager.cc:1367] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 08:11:16.411550 21926 consensus_queue.cc:260] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:16.412339 21934 raft_consensus.cc:348] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:16.412596 21934 raft_consensus.cc:374] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:16.412607 21926 ts_tablet_manager.cc:1398] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:16.412714 21934 raft_consensus.cc:729] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:16.413048 21934 consensus_queue.cc:260] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:16.413691 21934 ts_tablet_manager.cc:1398] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:16.460503 21940 raft_consensus.cc:482] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:16.460775 21940 raft_consensus.cc:504] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:16.463016 21940 leader_election.cc:288] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:16.483851 21941 raft_consensus.cc:482] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:16.484086 21941 raft_consensus.cc:504] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:16.486049 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3dfcb5d6c943bea49ebfb7ebb50796" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf" is_pre_election: true
I20240503 08:11:16.486303 21226 raft_consensus.cc:2454] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 0.
I20240503 08:11:16.486340 21941 leader_election.cc:288] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:16.487020 21423 leader_election.cc:302] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 3e04594b1a5344d9bfefa4de05d70842; no voters:
I20240503 08:11:16.486959 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3dfcb5d6c943bea49ebfb7ebb50796" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415" is_pre_election: true
I20240503 08:11:16.487442 21750 raft_consensus.cc:2454] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 0.
I20240503 08:11:16.487524 21940 raft_consensus.cc:2781] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:16.487665 21940 raft_consensus.cc:482] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:16.487768 21940 raft_consensus.cc:3037] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.494223 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3818719d4dc64f60b25efdc49c7e602b" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415" is_pre_election: true
I20240503 08:11:16.494493 21750 raft_consensus.cc:2454] T 3818719d4dc64f60b25efdc49c7e602b P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 0.
I20240503 08:11:16.494973 21162 leader_election.cc:302] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:16.495414 21941 raft_consensus.cc:2781] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:16.495533 21941 raft_consensus.cc:482] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:16.495668 21941 raft_consensus.cc:3037] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.496011 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3818719d4dc64f60b25efdc49c7e602b" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842" is_pre_election: true
I20240503 08:11:16.496256 21488 raft_consensus.cc:2454] T 3818719d4dc64f60b25efdc49c7e602b P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 0.
I20240503 08:11:16.574087 21940 raft_consensus.cc:504] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:16.574087 21941 raft_consensus.cc:504] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:16.574939 21941 leader_election.cc:288] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 election: Requested vote from peers 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:16.574966 21940 leader_election.cc:288] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:16.575481 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3dfcb5d6c943bea49ebfb7ebb50796" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf"
I20240503 08:11:16.575771 21226 raft_consensus.cc:3037] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.575795 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3818719d4dc64f60b25efdc49c7e602b" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415"
I20240503 08:11:16.575922 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3818719d4dc64f60b25efdc49c7e602b" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842"
I20240503 08:11:16.576028 21750 raft_consensus.cc:3037] T 3818719d4dc64f60b25efdc49c7e602b P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.576146 21488 raft_consensus.cc:3037] T 3818719d4dc64f60b25efdc49c7e602b P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.578780 21226 raft_consensus.cc:2454] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:16.578969 21488 raft_consensus.cc:2454] T 3818719d4dc64f60b25efdc49c7e602b P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 1.
I20240503 08:11:16.579494 21423 leader_election.cc:302] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 3e04594b1a5344d9bfefa4de05d70842; no voters:
I20240503 08:11:16.579667 21749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3dfcb5d6c943bea49ebfb7ebb50796" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415"
I20240503 08:11:16.579857 21160 leader_election.cc:302] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 3e04594b1a5344d9bfefa4de05d70842; no voters:
I20240503 08:11:16.579924 21940 raft_consensus.cc:2781] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:16.579977 21749 raft_consensus.cc:3037] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.580178 21941 raft_consensus.cc:2781] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:16.580699 21940 raft_consensus.cc:686] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 LEADER]: Becoming Leader. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Running, Role: LEADER
I20240503 08:11:16.580745 21941 raft_consensus.cc:686] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [term 1 LEADER]: Becoming Leader. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Running, Role: LEADER
I20240503 08:11:16.581277 21940 consensus_queue.cc:237] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:16.581300 21941 consensus_queue.cc:237] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:16.584889 20949 catalog_manager.cc:5484] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132). New cstate: current_term: 1 leader_uuid: "3e04594b1a5344d9bfefa4de05d70842" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:16.585269 20952 catalog_manager.cc:5484] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130). New cstate: current_term: 1 leader_uuid: "3b6eea641d214c79988ec6ad8821a4bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:16.622107 21935 raft_consensus.cc:482] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:16.622332 21935 raft_consensus.cc:504] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:16.623831 21935 leader_election.cc:288] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:16.632681 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d192acaf3cf74b508944391d49c95cbc" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" is_pre_election: true
I20240503 08:11:16.632977 21619 raft_consensus.cc:2454] T d192acaf3cf74b508944391d49c95cbc P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 0.
I20240503 08:11:16.633080 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d192acaf3cf74b508944391d49c95cbc" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd" is_pre_election: true
I20240503 08:11:16.633457 21094 raft_consensus.cc:2454] T d192acaf3cf74b508944391d49c95cbc P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 0.
I20240503 08:11:16.633494 21292 leader_election.cc:302] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:16.633927 21935 raft_consensus.cc:2781] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:16.634059 21935 raft_consensus.cc:482] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:16.634166 21935 raft_consensus.cc:3037] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.675235 21750 raft_consensus.cc:2454] T 3818719d4dc64f60b25efdc49c7e602b P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 1.
I20240503 08:11:16.675231 21935 raft_consensus.cc:504] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:16.676113 21935 leader_election.cc:288] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 election: Requested vote from peers 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:16.676851 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d192acaf3cf74b508944391d49c95cbc" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
I20240503 08:11:16.677024 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d192acaf3cf74b508944391d49c95cbc" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd"
I20240503 08:11:16.677119 21619 raft_consensus.cc:3037] T d192acaf3cf74b508944391d49c95cbc P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.677229 21094 raft_consensus.cc:3037] T d192acaf3cf74b508944391d49c95cbc P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.677716 21749 raft_consensus.cc:2454] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:16.774369 21619 raft_consensus.cc:2454] T d192acaf3cf74b508944391d49c95cbc P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 1.
I20240503 08:11:16.774401 21094 raft_consensus.cc:2454] T d192acaf3cf74b508944391d49c95cbc P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 1.
I20240503 08:11:16.775077 21292 leader_election.cc:302] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:16.775439 21935 raft_consensus.cc:2781] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:16.775861 21935 raft_consensus.cc:686] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 LEADER]: Becoming Leader. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Running, Role: LEADER
I20240503 08:11:16.776367 21935 consensus_queue.cc:237] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:16.780184 20952 catalog_manager.cc:5484] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131). New cstate: current_term: 1 leader_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:17.003306 21940 consensus_queue.cc:1035] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:17.008929 21940 consensus_queue.cc:1035] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:17.106598 21941 consensus_queue.cc:1035] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:17.115257 21941 consensus_queue.cc:1035] T 3818719d4dc64f60b25efdc49c7e602b P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:17.164582 21964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:17.164992 21964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:17.173146 21964 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:11:17.179234 21935 consensus_queue.cc:1035] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:17.185489 21935 consensus_queue.cc:1035] T d192acaf3cf74b508944391d49c95cbc P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 238.392 ms
time per row: 0.119196 ms
I20240503 08:11:17.604708 20952 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41046:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:17.616216 21074 tablet_service.cc:1467] Processing CreateTablet for tablet 533a5cf7eba34035a066a47862b54445 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7fdb9a57595c48d58d4e6abeac3059df]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.616809 21074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533a5cf7eba34035a066a47862b54445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.617066 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 97fd5f8f822f49b49f18cad0989c1c7e (DEFAULT_TABLE table=rebalance_test_table_2 [id=7fdb9a57595c48d58d4e6abeac3059df]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.617480 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97fd5f8f822f49b49f18cad0989c1c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.618036 21206 tablet_service.cc:1467] Processing CreateTablet for tablet caf143a06fd3435fa538f1550bce641f (DEFAULT_TABLE table=rebalance_test_table_2 [id=7fdb9a57595c48d58d4e6abeac3059df]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.618440 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf143a06fd3435fa538f1550bce641f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.621387 21598 tablet_service.cc:1467] Processing CreateTablet for tablet 533a5cf7eba34035a066a47862b54445 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7fdb9a57595c48d58d4e6abeac3059df]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.621840 21598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533a5cf7eba34035a066a47862b54445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.625368 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 533a5cf7eba34035a066a47862b54445 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7fdb9a57595c48d58d4e6abeac3059df]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.625813 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533a5cf7eba34035a066a47862b54445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.625924 21730 tablet_service.cc:1467] Processing CreateTablet for tablet caf143a06fd3435fa538f1550bce641f (DEFAULT_TABLE table=rebalance_test_table_2 [id=7fdb9a57595c48d58d4e6abeac3059df]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.626439 21730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf143a06fd3435fa538f1550bce641f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.629509 21467 tablet_service.cc:1467] Processing CreateTablet for tablet caf143a06fd3435fa538f1550bce641f (DEFAULT_TABLE table=rebalance_test_table_2 [id=7fdb9a57595c48d58d4e6abeac3059df]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.630024 21467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf143a06fd3435fa538f1550bce641f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.629954 21073 tablet_service.cc:1467] Processing CreateTablet for tablet 97fd5f8f822f49b49f18cad0989c1c7e (DEFAULT_TABLE table=rebalance_test_table_2 [id=7fdb9a57595c48d58d4e6abeac3059df]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.630468 21073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97fd5f8f822f49b49f18cad0989c1c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.633783 21599 tablet_service.cc:1467] Processing CreateTablet for tablet 97fd5f8f822f49b49f18cad0989c1c7e (DEFAULT_TABLE table=rebalance_test_table_2 [id=7fdb9a57595c48d58d4e6abeac3059df]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.634006 22032 tablet_bootstrap.cc:492] T 97fd5f8f822f49b49f18cad0989c1c7e P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:17.634260 21599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97fd5f8f822f49b49f18cad0989c1c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.639580 22032 tablet_bootstrap.cc:654] T 97fd5f8f822f49b49f18cad0989c1c7e P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.651965 22036 tablet_bootstrap.cc:492] T 97fd5f8f822f49b49f18cad0989c1c7e P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:17.657266 22036 tablet_bootstrap.cc:654] T 97fd5f8f822f49b49f18cad0989c1c7e P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.660130 22035 tablet_bootstrap.cc:492] T caf143a06fd3435fa538f1550bce641f P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:17.660615 22033 tablet_bootstrap.cc:492] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:17.664638 22038 tablet_bootstrap.cc:492] T caf143a06fd3435fa538f1550bce641f P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:17.665282 22034 tablet_bootstrap.cc:492] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:17.666263 22035 tablet_bootstrap.cc:654] T caf143a06fd3435fa538f1550bce641f P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.668264 22033 tablet_bootstrap.cc:654] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.670498 22034 tablet_bootstrap.cc:654] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.671923 22032 tablet_bootstrap.cc:492] T 97fd5f8f822f49b49f18cad0989c1c7e P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:17.671972 22038 tablet_bootstrap.cc:654] T caf143a06fd3435fa538f1550bce641f P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.672169 22032 ts_tablet_manager.cc:1367] T 97fd5f8f822f49b49f18cad0989c1c7e P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.007s
I20240503 08:11:17.673544 22032 raft_consensus.cc:348] T 97fd5f8f822f49b49f18cad0989c1c7e P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:17.673990 22032 raft_consensus.cc:374] T 97fd5f8f822f49b49f18cad0989c1c7e P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.674167 22032 raft_consensus.cc:729] T 97fd5f8f822f49b49f18cad0989c1c7e P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.674666 22032 consensus_queue.cc:260] T 97fd5f8f822f49b49f18cad0989c1c7e P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:17.679126 22032 ts_tablet_manager.cc:1398] T 97fd5f8f822f49b49f18cad0989c1c7e P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:11:17.683801 22032 tablet_bootstrap.cc:492] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:17.685621 22034 tablet_bootstrap.cc:492] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:17.686719 22038 tablet_bootstrap.cc:492] T caf143a06fd3435fa538f1550bce641f P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:17.687269 22034 ts_tablet_manager.cc:1367] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20240503 08:11:17.687330 22032 tablet_bootstrap.cc:654] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.687507 22038 ts_tablet_manager.cc:1367] T caf143a06fd3435fa538f1550bce641f P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.007s
I20240503 08:11:17.688529 22034 raft_consensus.cc:348] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:17.689782 22038 raft_consensus.cc:348] T caf143a06fd3435fa538f1550bce641f P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:17.690096 22034 raft_consensus.cc:374] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.690200 22036 tablet_bootstrap.cc:492] T 97fd5f8f822f49b49f18cad0989c1c7e P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:17.690260 22034 raft_consensus.cc:729] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.690419 22038 raft_consensus.cc:374] T caf143a06fd3435fa538f1550bce641f P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.690469 22036 ts_tablet_manager.cc:1367] T 97fd5f8f822f49b49f18cad0989c1c7e P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.003s
I20240503 08:11:17.690703 22038 raft_consensus.cc:729] T caf143a06fd3435fa538f1550bce641f P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.690968 22034 consensus_queue.cc:260] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:17.691526 22038 consensus_queue.cc:260] T caf143a06fd3435fa538f1550bce641f P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:17.691637 22036 raft_consensus.cc:348] T 97fd5f8f822f49b49f18cad0989c1c7e P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:17.691991 22036 raft_consensus.cc:374] T 97fd5f8f822f49b49f18cad0989c1c7e P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.692163 22036 raft_consensus.cc:729] T 97fd5f8f822f49b49f18cad0989c1c7e P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.692541 22036 consensus_queue.cc:260] T 97fd5f8f822f49b49f18cad0989c1c7e P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:17.695024 22036 ts_tablet_manager.cc:1398] T 97fd5f8f822f49b49f18cad0989c1c7e P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:11:17.695513 22036 tablet_bootstrap.cc:492] T 533a5cf7eba34035a066a47862b54445 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:17.697921 22034 ts_tablet_manager.cc:1398] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 08:11:17.698436 22038 ts_tablet_manager.cc:1398] T caf143a06fd3435fa538f1550bce641f P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:11:17.698467 22032 tablet_bootstrap.cc:492] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:17.698637 22032 ts_tablet_manager.cc:1367] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20240503 08:11:17.699532 22032 raft_consensus.cc:348] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:17.701004 22036 tablet_bootstrap.cc:654] T 533a5cf7eba34035a066a47862b54445 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.701165 22032 raft_consensus.cc:374] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.701299 22032 raft_consensus.cc:729] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.701575 22033 tablet_bootstrap.cc:492] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:17.701706 22035 tablet_bootstrap.cc:492] T caf143a06fd3435fa538f1550bce641f P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:17.701633 22032 consensus_queue.cc:260] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:17.701817 22033 ts_tablet_manager.cc:1367] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.004s
I20240503 08:11:17.701910 22035 ts_tablet_manager.cc:1367] T caf143a06fd3435fa538f1550bce641f P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.029s
I20240503 08:11:17.702796 22035 raft_consensus.cc:348] T caf143a06fd3435fa538f1550bce641f P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:17.702843 22033 raft_consensus.cc:348] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:17.703052 22035 raft_consensus.cc:374] T caf143a06fd3435fa538f1550bce641f P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.703176 22035 raft_consensus.cc:729] T caf143a06fd3435fa538f1550bce641f P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.703193 22033 raft_consensus.cc:374] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.703374 22033 raft_consensus.cc:729] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.703492 22035 consensus_queue.cc:260] T caf143a06fd3435fa538f1550bce641f P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:17.703760 22033 consensus_queue.cc:260] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:17.704116 22035 ts_tablet_manager.cc:1398] T caf143a06fd3435fa538f1550bce641f P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:17.704625 22033 ts_tablet_manager.cc:1398] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:11:17.705338 22032 ts_tablet_manager.cc:1398] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 08:11:17.707784 22034 tablet_bootstrap.cc:492] T 533a5cf7eba34035a066a47862b54445 P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:17.710244 22034 tablet_bootstrap.cc:654] T 533a5cf7eba34035a066a47862b54445 P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.712447 22034 tablet_bootstrap.cc:492] T 533a5cf7eba34035a066a47862b54445 P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:17.712621 22034 ts_tablet_manager.cc:1367] T 533a5cf7eba34035a066a47862b54445 P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:11:17.712742 22036 tablet_bootstrap.cc:492] T 533a5cf7eba34035a066a47862b54445 P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:17.712937 22036 ts_tablet_manager.cc:1367] T 533a5cf7eba34035a066a47862b54445 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.002s
I20240503 08:11:17.713493 22034 raft_consensus.cc:348] T 533a5cf7eba34035a066a47862b54445 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:17.713757 22034 raft_consensus.cc:374] T 533a5cf7eba34035a066a47862b54445 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.713850 22034 raft_consensus.cc:729] T 533a5cf7eba34035a066a47862b54445 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.713701 22036 raft_consensus.cc:348] T 533a5cf7eba34035a066a47862b54445 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:17.714004 22036 raft_consensus.cc:374] T 533a5cf7eba34035a066a47862b54445 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.714100 22036 raft_consensus.cc:729] T 533a5cf7eba34035a066a47862b54445 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.714121 22034 consensus_queue.cc:260] T 533a5cf7eba34035a066a47862b54445 P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:17.714387 22036 consensus_queue.cc:260] T 533a5cf7eba34035a066a47862b54445 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:17.714754 22034 ts_tablet_manager.cc:1398] T 533a5cf7eba34035a066a47862b54445 P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:17.714977 22036 ts_tablet_manager.cc:1398] T 533a5cf7eba34035a066a47862b54445 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:17.716039 21962 raft_consensus.cc:482] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:17.716200 21962 raft_consensus.cc:504] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:17.716984 21962 leader_election.cc:288] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:17.717568 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533a5cf7eba34035a066a47862b54445" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd" is_pre_election: true
I20240503 08:11:17.717792 21094 raft_consensus.cc:2454] T 533a5cf7eba34035a066a47862b54445 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 0.
I20240503 08:11:17.718616 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533a5cf7eba34035a066a47862b54445" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" is_pre_election: true
I20240503 08:11:17.718783 21291 leader_election.cc:302] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d9776f5d634ec2b167dc225659e4fd, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:17.718845 21619 raft_consensus.cc:2454] T 533a5cf7eba34035a066a47862b54445 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 0.
I20240503 08:11:17.719681 21961 raft_consensus.cc:482] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:17.719846 21961 raft_consensus.cc:504] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:17.720053 21962 raft_consensus.cc:2781] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:17.720170 21962 raft_consensus.cc:482] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:17.720276 21962 raft_consensus.cc:3037] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.720613 21961 leader_election.cc:288] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:17.721099 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97fd5f8f822f49b49f18cad0989c1c7e" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" is_pre_election: true
I20240503 08:11:17.721370 21357 raft_consensus.cc:2454] T 97fd5f8f822f49b49f18cad0989c1c7e P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 0.
I20240503 08:11:17.721879 21027 leader_election.cc:302] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d9776f5d634ec2b167dc225659e4fd, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:17.722178 21961 raft_consensus.cc:2781] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:17.722288 21961 raft_consensus.cc:482] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:17.722362 21961 raft_consensus.cc:3037] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.723373 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97fd5f8f822f49b49f18cad0989c1c7e" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" is_pre_election: true
I20240503 08:11:17.723591 21619 raft_consensus.cc:2454] T 97fd5f8f822f49b49f18cad0989c1c7e P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 0.
I20240503 08:11:17.814872 21962 raft_consensus.cc:504] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:17.814880 21961 raft_consensus.cc:504] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:17.815634 21962 leader_election.cc:288] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 election: Requested vote from peers 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:17.815721 21961 leader_election.cc:288] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 election: Requested vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:17.816069 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533a5cf7eba34035a066a47862b54445" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd"
I20240503 08:11:17.816300 21094 raft_consensus.cc:3037] T 533a5cf7eba34035a066a47862b54445 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.816439 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97fd5f8f822f49b49f18cad0989c1c7e" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
I20240503 08:11:17.816674 21619 raft_consensus.cc:3037] T 97fd5f8f822f49b49f18cad0989c1c7e P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.816687 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97fd5f8f822f49b49f18cad0989c1c7e" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5"
I20240503 08:11:17.816887 21356 raft_consensus.cc:3037] T 97fd5f8f822f49b49f18cad0989c1c7e P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.819577 21619 raft_consensus.cc:2454] T 97fd5f8f822f49b49f18cad0989c1c7e P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 1.
I20240503 08:11:17.819681 21356 raft_consensus.cc:2454] T 97fd5f8f822f49b49f18cad0989c1c7e P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 1.
I20240503 08:11:17.819712 21094 raft_consensus.cc:2454] T 533a5cf7eba34035a066a47862b54445 P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 1.
I20240503 08:11:17.820107 21027 leader_election.cc:302] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 18d9776f5d634ec2b167dc225659e4fd, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:17.820406 21291 leader_election.cc:302] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d9776f5d634ec2b167dc225659e4fd, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:17.820437 21961 raft_consensus.cc:2781] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:17.820569 21961 raft_consensus.cc:686] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [term 1 LEADER]: Becoming Leader. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Running, Role: LEADER
I20240503 08:11:17.820981 21962 raft_consensus.cc:2781] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:17.820909 21961 consensus_queue.cc:237] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:17.821105 21962 raft_consensus.cc:686] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 LEADER]: Becoming Leader. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Running, Role: LEADER
I20240503 08:11:17.821391 21962 consensus_queue.cc:237] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:17.826380 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533a5cf7eba34035a066a47862b54445" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
I20240503 08:11:17.826597 21619 raft_consensus.cc:3037] T 533a5cf7eba34035a066a47862b54445 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.831040 20945 catalog_manager.cc:5484] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131). New cstate: current_term: 1 leader_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:17.833597 20949 catalog_manager.cc:5484] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129). New cstate: current_term: 1 leader_uuid: "18d9776f5d634ec2b167dc225659e4fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:17.854588 21972 raft_consensus.cc:482] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:17.854812 21972 raft_consensus.cc:504] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:17.855669 21972 leader_election.cc:288] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6503743925434401a74aae303628e415 (127.20.107.134:42791), 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603)
I20240503 08:11:17.856190 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf143a06fd3435fa538f1550bce641f" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842" is_pre_election: true
I20240503 08:11:17.856212 21749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf143a06fd3435fa538f1550bce641f" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415" is_pre_election: true
I20240503 08:11:17.856410 21488 raft_consensus.cc:2454] T caf143a06fd3435fa538f1550bce641f P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 0.
I20240503 08:11:17.856442 21749 raft_consensus.cc:2454] T caf143a06fd3435fa538f1550bce641f P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 0.
I20240503 08:11:17.856844 21160 leader_election.cc:302] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 3e04594b1a5344d9bfefa4de05d70842; no voters:
I20240503 08:11:17.857174 21972 raft_consensus.cc:2781] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:17.857286 21972 raft_consensus.cc:482] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:17.857378 21972 raft_consensus.cc:3037] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.916545 21619 raft_consensus.cc:2454] T 533a5cf7eba34035a066a47862b54445 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 1.
I20240503 08:11:17.916584 21972 raft_consensus.cc:504] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:17.917436 21972 leader_election.cc:288] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 election: Requested vote from peers 6503743925434401a74aae303628e415 (127.20.107.134:42791), 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603)
I20240503 08:11:17.917912 21749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf143a06fd3435fa538f1550bce641f" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415"
I20240503 08:11:17.918125 21749 raft_consensus.cc:3037] T caf143a06fd3435fa538f1550bce641f P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.918045 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf143a06fd3435fa538f1550bce641f" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842"
I20240503 08:11:17.918275 21488 raft_consensus.cc:3037] T caf143a06fd3435fa538f1550bce641f P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.920089 21749 raft_consensus.cc:2454] T caf143a06fd3435fa538f1550bce641f P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 1.
I20240503 08:11:17.920532 21162 leader_election.cc:302] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:17.920833 21972 raft_consensus.cc:2781] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:17.921007 21972 raft_consensus.cc:686] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [term 1 LEADER]: Becoming Leader. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Running, Role: LEADER
I20240503 08:11:17.921346 21972 consensus_queue.cc:237] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:17.924278 20949 catalog_manager.cc:5484] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130). New cstate: current_term: 1 leader_uuid: "3b6eea641d214c79988ec6ad8821a4bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:18.021644 21488 raft_consensus.cc:2454] T caf143a06fd3435fa538f1550bce641f P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 1.
I20240503 08:11:18.205257 21962 consensus_queue.cc:1035] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:18.226418 21981 consensus_queue.cc:1035] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:18.233036 21961 consensus_queue.cc:1035] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:18.238770 22048 consensus_queue.cc:1035] T 97fd5f8f822f49b49f18cad0989c1c7e P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:18.306043 22052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:18.306412 22052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:18.314365 22052 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 08:11:18.465402 21972 consensus_queue.cc:1035] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:18.496332 21749 raft_consensus.cc:1261] T caf143a06fd3435fa538f1550bce641f P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Refusing update from remote peer 3b6eea641d214c79988ec6ad8821a4bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:18.499508 21972 consensus_queue.cc:1035] T caf143a06fd3435fa538f1550bce641f P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:18.563838 22107 mvcc.cc:205] Tried to move back new op lower bound from 7023509006217252864 to 7023509003968049152. Current Snapshot: MvccSnapshot[applied={T|T < 7023509006217252864}]
INSERT report
rows total: 2000
time total: 219.177 ms
time per row: 0.109588 ms
I20240503 08:11:18.697271 20952 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41046:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:18.706424 21467 tablet_service.cc:1467] Processing CreateTablet for tablet 6d23a8d6a1a2476b82d441f5d289b9fb (DEFAULT_TABLE table=rebalance_test_table_3 [id=a21c891a52b24605831a383af4d7fb0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.706558 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 6d23a8d6a1a2476b82d441f5d289b9fb (DEFAULT_TABLE table=rebalance_test_table_3 [id=a21c891a52b24605831a383af4d7fb0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.706972 21467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d23a8d6a1a2476b82d441f5d289b9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.706981 21730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d23a8d6a1a2476b82d441f5d289b9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.708106 21337 tablet_service.cc:1467] Processing CreateTablet for tablet e5cdd453d491408095e460eb78265e03 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a21c891a52b24605831a383af4d7fb0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.707396 21729 tablet_service.cc:1467] Processing CreateTablet for tablet fbf2ad4b08ec4646896e6add57ca3caa (DEFAULT_TABLE table=rebalance_test_table_3 [id=a21c891a52b24605831a383af4d7fb0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.708501 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5cdd453d491408095e460eb78265e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.708401 21073 tablet_service.cc:1467] Processing CreateTablet for tablet e5cdd453d491408095e460eb78265e03 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a21c891a52b24605831a383af4d7fb0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.708628 21729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf2ad4b08ec4646896e6add57ca3caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.708829 21073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5cdd453d491408095e460eb78265e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.712699 21598 tablet_service.cc:1467] Processing CreateTablet for tablet e5cdd453d491408095e460eb78265e03 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a21c891a52b24605831a383af4d7fb0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.713137 21598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5cdd453d491408095e460eb78265e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.717653 21468 tablet_service.cc:1467] Processing CreateTablet for tablet fbf2ad4b08ec4646896e6add57ca3caa (DEFAULT_TABLE table=rebalance_test_table_3 [id=a21c891a52b24605831a383af4d7fb0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.717849 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 6d23a8d6a1a2476b82d441f5d289b9fb (DEFAULT_TABLE table=rebalance_test_table_3 [id=a21c891a52b24605831a383af4d7fb0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.718115 21468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf2ad4b08ec4646896e6add57ca3caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.718406 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d23a8d6a1a2476b82d441f5d289b9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.723718 22123 tablet_bootstrap.cc:492] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:18.725302 21205 tablet_service.cc:1467] Processing CreateTablet for tablet fbf2ad4b08ec4646896e6add57ca3caa (DEFAULT_TABLE table=rebalance_test_table_3 [id=a21c891a52b24605831a383af4d7fb0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.725426 22123 tablet_bootstrap.cc:654] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.725725 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf2ad4b08ec4646896e6add57ca3caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.726745 22124 tablet_bootstrap.cc:492] T e5cdd453d491408095e460eb78265e03 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:18.728480 22124 tablet_bootstrap.cc:654] T e5cdd453d491408095e460eb78265e03 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.731905 22126 tablet_bootstrap.cc:492] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:18.735289 22128 tablet_bootstrap.cc:492] T fbf2ad4b08ec4646896e6add57ca3caa P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:18.736506 22126 tablet_bootstrap.cc:654] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.736742 22124 tablet_bootstrap.cc:492] T e5cdd453d491408095e460eb78265e03 P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:18.736984 22124 ts_tablet_manager.cc:1367] T e5cdd453d491408095e460eb78265e03 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 08:11:18.737874 22128 tablet_bootstrap.cc:654] T fbf2ad4b08ec4646896e6add57ca3caa P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.737910 22124 raft_consensus.cc:348] T e5cdd453d491408095e460eb78265e03 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:18.738219 22124 raft_consensus.cc:374] T e5cdd453d491408095e460eb78265e03 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.738368 22124 raft_consensus.cc:729] T e5cdd453d491408095e460eb78265e03 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.738735 22124 consensus_queue.cc:260] T e5cdd453d491408095e460eb78265e03 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:18.739979 22123 tablet_bootstrap.cc:492] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:18.740983 22123 ts_tablet_manager.cc:1367] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 08:11:18.742520 22122 tablet_bootstrap.cc:492] T fbf2ad4b08ec4646896e6add57ca3caa P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:18.742372 22123 raft_consensus.cc:348] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:18.742657 22123 raft_consensus.cc:374] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.742784 22123 raft_consensus.cc:729] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.743403 22121 tablet_bootstrap.cc:492] T e5cdd453d491408095e460eb78265e03 P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:18.743817 22123 consensus_queue.cc:260] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:18.743952 22124 ts_tablet_manager.cc:1398] T e5cdd453d491408095e460eb78265e03 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:11:18.744460 22123 ts_tablet_manager.cc:1398] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:18.744776 22123 tablet_bootstrap.cc:492] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:18.746570 22123 tablet_bootstrap.cc:654] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.747015 22121 tablet_bootstrap.cc:654] T e5cdd453d491408095e460eb78265e03 P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.747228 22122 tablet_bootstrap.cc:654] T fbf2ad4b08ec4646896e6add57ca3caa P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.748059 22128 tablet_bootstrap.cc:492] T fbf2ad4b08ec4646896e6add57ca3caa P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:18.748241 22128 ts_tablet_manager.cc:1367] T fbf2ad4b08ec4646896e6add57ca3caa P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240503 08:11:18.749162 22128 raft_consensus.cc:348] T fbf2ad4b08ec4646896e6add57ca3caa P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:18.750437 22128 raft_consensus.cc:374] T fbf2ad4b08ec4646896e6add57ca3caa P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.751451 22126 tablet_bootstrap.cc:492] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:18.751725 22126 ts_tablet_manager.cc:1367] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20240503 08:11:18.752785 22126 raft_consensus.cc:348] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:18.753103 22126 raft_consensus.cc:374] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.753216 22126 raft_consensus.cc:729] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.753250 22123 tablet_bootstrap.cc:492] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:18.753414 22123 ts_tablet_manager.cc:1367] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 08:11:18.753508 22126 consensus_queue.cc:260] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:18.750583 22128 raft_consensus.cc:729] T fbf2ad4b08ec4646896e6add57ca3caa P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.754122 22123 raft_consensus.cc:348] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:18.754398 22123 raft_consensus.cc:374] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.754297 22128 consensus_queue.cc:260] T fbf2ad4b08ec4646896e6add57ca3caa P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:18.754515 22123 raft_consensus.cc:729] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.754793 22123 consensus_queue.cc:260] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:18.755399 22123 ts_tablet_manager.cc:1398] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:18.756943 22126 ts_tablet_manager.cc:1398] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:11:18.757900 22122 tablet_bootstrap.cc:492] T fbf2ad4b08ec4646896e6add57ca3caa P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:18.758049 22122 ts_tablet_manager.cc:1367] T fbf2ad4b08ec4646896e6add57ca3caa P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.007s
I20240503 08:11:18.758134 22128 ts_tablet_manager.cc:1398] T fbf2ad4b08ec4646896e6add57ca3caa P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240503 08:11:18.758428 22128 tablet_bootstrap.cc:492] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:18.758834 22122 raft_consensus.cc:348] T fbf2ad4b08ec4646896e6add57ca3caa P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:18.759043 22122 raft_consensus.cc:374] T fbf2ad4b08ec4646896e6add57ca3caa P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.759131 22122 raft_consensus.cc:729] T fbf2ad4b08ec4646896e6add57ca3caa P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.759369 22122 consensus_queue.cc:260] T fbf2ad4b08ec4646896e6add57ca3caa P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:18.759898 22122 ts_tablet_manager.cc:1398] T fbf2ad4b08ec4646896e6add57ca3caa P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:11:18.760210 22122 tablet_bootstrap.cc:492] T 6d23a8d6a1a2476b82d441f5d289b9fb P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:18.761111 22128 tablet_bootstrap.cc:654] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.761850 22122 tablet_bootstrap.cc:654] T 6d23a8d6a1a2476b82d441f5d289b9fb P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.763161 22121 tablet_bootstrap.cc:492] T e5cdd453d491408095e460eb78265e03 P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:18.763409 22121 ts_tablet_manager.cc:1367] T e5cdd453d491408095e460eb78265e03 P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240503 08:11:18.764765 22121 raft_consensus.cc:348] T e5cdd453d491408095e460eb78265e03 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:18.765153 22121 raft_consensus.cc:374] T e5cdd453d491408095e460eb78265e03 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.765280 22121 raft_consensus.cc:729] T e5cdd453d491408095e460eb78265e03 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.765581 22121 consensus_queue.cc:260] T e5cdd453d491408095e460eb78265e03 P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:18.766342 22121 ts_tablet_manager.cc:1398] T e5cdd453d491408095e460eb78265e03 P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:11:18.767275 22122 tablet_bootstrap.cc:492] T 6d23a8d6a1a2476b82d441f5d289b9fb P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:18.767446 22122 ts_tablet_manager.cc:1367] T 6d23a8d6a1a2476b82d441f5d289b9fb P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240503 08:11:18.767493 22128 tablet_bootstrap.cc:492] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:18.767705 22128 ts_tablet_manager.cc:1367] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20240503 08:11:18.768121 22122 raft_consensus.cc:348] T 6d23a8d6a1a2476b82d441f5d289b9fb P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:18.768345 22122 raft_consensus.cc:374] T 6d23a8d6a1a2476b82d441f5d289b9fb P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.768431 22122 raft_consensus.cc:729] T 6d23a8d6a1a2476b82d441f5d289b9fb P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.768550 22128 raft_consensus.cc:348] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:18.768812 22128 raft_consensus.cc:374] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.768723 22122 consensus_queue.cc:260] T 6d23a8d6a1a2476b82d441f5d289b9fb P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:18.768956 22128 raft_consensus.cc:729] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.769333 22122 ts_tablet_manager.cc:1398] T 6d23a8d6a1a2476b82d441f5d289b9fb P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:18.769342 22128 consensus_queue.cc:260] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:18.769948 22128 ts_tablet_manager.cc:1398] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:18.800827 22103 raft_consensus.cc:482] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:18.801044 22103 raft_consensus.cc:504] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:18.801805 22103 leader_election.cc:288] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:18.802438 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d23a8d6a1a2476b82d441f5d289b9fb" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842" is_pre_election: true
I20240503 08:11:18.802438 21749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d23a8d6a1a2476b82d441f5d289b9fb" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415" is_pre_election: true
I20240503 08:11:18.802704 21488 raft_consensus.cc:2454] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 0.
I20240503 08:11:18.802693 21749 raft_consensus.cc:2454] T 6d23a8d6a1a2476b82d441f5d289b9fb P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 0.
I20240503 08:11:18.803228 21160 leader_election.cc:302] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 3e04594b1a5344d9bfefa4de05d70842; no voters:
I20240503 08:11:18.803556 22103 raft_consensus.cc:2781] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:18.803676 22103 raft_consensus.cc:482] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:18.803771 22103 raft_consensus.cc:3037] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.836182 22103 raft_consensus.cc:504] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:18.837000 22103 leader_election.cc:288] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 election: Requested vote from peers 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:18.837759 21749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d23a8d6a1a2476b82d441f5d289b9fb" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415"
I20240503 08:11:18.837983 21749 raft_consensus.cc:3037] T 6d23a8d6a1a2476b82d441f5d289b9fb P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.838106 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d23a8d6a1a2476b82d441f5d289b9fb" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842"
I20240503 08:11:18.838346 21488 raft_consensus.cc:3037] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.840740 21749 raft_consensus.cc:2454] T 6d23a8d6a1a2476b82d441f5d289b9fb P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 1.
I20240503 08:11:18.841282 21162 leader_election.cc:302] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:18.841619 22103 raft_consensus.cc:2781] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:18.841763 22103 raft_consensus.cc:686] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [term 1 LEADER]: Becoming Leader. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Running, Role: LEADER
I20240503 08:11:18.842031 22103 consensus_queue.cc:237] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:18.847007 20947 catalog_manager.cc:5484] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130). New cstate: current_term: 1 leader_uuid: "3b6eea641d214c79988ec6ad8821a4bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:18.876245 21940 raft_consensus.cc:482] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:18.876466 21940 raft_consensus.cc:504] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:18.877219 21940 leader_election.cc:288] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:18.877743 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf2ad4b08ec4646896e6add57ca3caa" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf" is_pre_election: true
I20240503 08:11:18.877801 21749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf2ad4b08ec4646896e6add57ca3caa" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415" is_pre_election: true
I20240503 08:11:18.877943 21226 raft_consensus.cc:2454] T fbf2ad4b08ec4646896e6add57ca3caa P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 0.
I20240503 08:11:18.878036 21749 raft_consensus.cc:2454] T fbf2ad4b08ec4646896e6add57ca3caa P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 0.
I20240503 08:11:18.878338 21423 leader_election.cc:302] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 3e04594b1a5344d9bfefa4de05d70842; no voters:
I20240503 08:11:18.878580 21940 raft_consensus.cc:2781] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:18.878664 21940 raft_consensus.cc:482] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:18.878760 21940 raft_consensus.cc:3037] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.903167 22134 raft_consensus.cc:482] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:18.903417 22134 raft_consensus.cc:504] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:18.905079 22134 leader_election.cc:288] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921), 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537)
I20240503 08:11:18.912361 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cdd453d491408095e460eb78265e03" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd" is_pre_election: true
I20240503 08:11:18.912353 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cdd453d491408095e460eb78265e03" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" is_pre_election: true
I20240503 08:11:18.912616 21094 raft_consensus.cc:2454] T e5cdd453d491408095e460eb78265e03 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 0.
I20240503 08:11:18.912632 21356 raft_consensus.cc:2454] T e5cdd453d491408095e460eb78265e03 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 0.
I20240503 08:11:18.913203 21553 leader_election.cc:302] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 18d9776f5d634ec2b167dc225659e4fd; no voters:
I20240503 08:11:18.913605 22134 raft_consensus.cc:2781] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:18.913748 22134 raft_consensus.cc:482] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:18.913858 22134 raft_consensus.cc:3037] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.939021 21488 raft_consensus.cc:2454] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 1.
I20240503 08:11:18.939129 22134 raft_consensus.cc:504] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:18.940311 22134 leader_election.cc:288] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 election: Requested vote from peers 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921), 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537)
I20240503 08:11:18.940840 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cdd453d491408095e460eb78265e03" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd"
I20240503 08:11:18.941103 21094 raft_consensus.cc:3037] T e5cdd453d491408095e460eb78265e03 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.941074 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cdd453d491408095e460eb78265e03" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5"
I20240503 08:11:18.941306 21356 raft_consensus.cc:3037] T e5cdd453d491408095e460eb78265e03 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.941411 21940 raft_consensus.cc:504] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:18.942534 21749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf2ad4b08ec4646896e6add57ca3caa" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415"
I20240503 08:11:18.942734 21749 raft_consensus.cc:3037] T fbf2ad4b08ec4646896e6add57ca3caa P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.943619 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf2ad4b08ec4646896e6add57ca3caa" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf"
I20240503 08:11:18.943833 21226 raft_consensus.cc:3037] T fbf2ad4b08ec4646896e6add57ca3caa P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.944072 21940 leader_election.cc:288] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:19.041620 21356 raft_consensus.cc:2454] T e5cdd453d491408095e460eb78265e03 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 1.
I20240503 08:11:19.041638 21749 raft_consensus.cc:2454] T fbf2ad4b08ec4646896e6add57ca3caa P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:19.041620 21094 raft_consensus.cc:2454] T e5cdd453d491408095e460eb78265e03 P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 1.
I20240503 08:11:19.042290 21424 leader_election.cc:302] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e04594b1a5344d9bfefa4de05d70842, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:19.042277 21553 leader_election.cc:302] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 18d9776f5d634ec2b167dc225659e4fd; no voters:
I20240503 08:11:19.042315 21226 raft_consensus.cc:2454] T fbf2ad4b08ec4646896e6add57ca3caa P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:19.042709 21940 raft_consensus.cc:2781] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:19.042850 22134 raft_consensus.cc:2781] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:19.043222 21940 raft_consensus.cc:686] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [term 1 LEADER]: Becoming Leader. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Running, Role: LEADER
I20240503 08:11:19.044080 22134 raft_consensus.cc:686] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 LEADER]: Becoming Leader. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Running, Role: LEADER
I20240503 08:11:19.043697 21940 consensus_queue.cc:237] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:19.044831 22134 consensus_queue.cc:237] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:19.047544 20947 catalog_manager.cc:5484] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132). New cstate: current_term: 1 leader_uuid: "3e04594b1a5344d9bfefa4de05d70842" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:19.048717 20952 catalog_manager.cc:5484] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133). New cstate: current_term: 1 leader_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:19.269672 22103 consensus_queue.cc:1035] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:19.278270 22102 consensus_queue.cc:1035] T 6d23a8d6a1a2476b82d441f5d289b9fb P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:19.447764 22144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:19.448096 22144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:19.456106 22144 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:11:19.476644 21940 consensus_queue.cc:1035] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:19.481920 21940 consensus_queue.cc:1035] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240503 08:11:19.563207 22175 consensus_queue.cc:1035] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:19.593122 22177 consensus_queue.cc:1035] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 247.512 ms
time per row: 0.123756 ms
I20240503 08:11:19.888792 20952 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41046:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:19.897392 21073 tablet_service.cc:1467] Processing CreateTablet for tablet 977a045f5f4b469f846aad8acf86b0b0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cdde13f8ec074bf4bd701f7b9f59d63e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.897617 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 977a045f5f4b469f846aad8acf86b0b0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cdde13f8ec074bf4bd701f7b9f59d63e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.897769 21598 tablet_service.cc:1467] Processing CreateTablet for tablet 977a045f5f4b469f846aad8acf86b0b0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cdde13f8ec074bf4bd701f7b9f59d63e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.897877 21073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 977a045f5f4b469f846aad8acf86b0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.898064 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 977a045f5f4b469f846aad8acf86b0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.898121 21598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 977a045f5f4b469f846aad8acf86b0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.899430 21599 tablet_service.cc:1467] Processing CreateTablet for tablet 08c639a95ae940808c97c5dfcdc13caf (DEFAULT_TABLE table=rebalance_test_table_4 [id=cdde13f8ec074bf4bd701f7b9f59d63e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.899804 21599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c639a95ae940808c97c5dfcdc13caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.901610 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 08c639a95ae940808c97c5dfcdc13caf (DEFAULT_TABLE table=rebalance_test_table_4 [id=cdde13f8ec074bf4bd701f7b9f59d63e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.901957 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c639a95ae940808c97c5dfcdc13caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.904228 21205 tablet_service.cc:1467] Processing CreateTablet for tablet 64a39d113638472580a5932bcc8531a3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cdde13f8ec074bf4bd701f7b9f59d63e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.904774 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a39d113638472580a5932bcc8531a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.909160 22217 tablet_bootstrap.cc:492] T 08c639a95ae940808c97c5dfcdc13caf P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:19.910341 21729 tablet_service.cc:1467] Processing CreateTablet for tablet 64a39d113638472580a5932bcc8531a3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cdde13f8ec074bf4bd701f7b9f59d63e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.910926 21729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a39d113638472580a5932bcc8531a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.911849 22217 tablet_bootstrap.cc:654] T 08c639a95ae940808c97c5dfcdc13caf P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.913596 22215 tablet_bootstrap.cc:492] T 977a045f5f4b469f846aad8acf86b0b0 P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:19.913691 21074 tablet_service.cc:1467] Processing CreateTablet for tablet 08c639a95ae940808c97c5dfcdc13caf (DEFAULT_TABLE table=rebalance_test_table_4 [id=cdde13f8ec074bf4bd701f7b9f59d63e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.914127 21074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c639a95ae940808c97c5dfcdc13caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.915946 22215 tablet_bootstrap.cc:654] T 977a045f5f4b469f846aad8acf86b0b0 P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.917309 21467 tablet_service.cc:1467] Processing CreateTablet for tablet 64a39d113638472580a5932bcc8531a3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cdde13f8ec074bf4bd701f7b9f59d63e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.917723 21467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a39d113638472580a5932bcc8531a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.924459 22217 tablet_bootstrap.cc:492] T 08c639a95ae940808c97c5dfcdc13caf P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:19.925010 22215 tablet_bootstrap.cc:492] T 977a045f5f4b469f846aad8acf86b0b0 P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:19.925242 22215 ts_tablet_manager.cc:1367] T 977a045f5f4b469f846aad8acf86b0b0 P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 08:11:19.925238 22217 ts_tablet_manager.cc:1367] T 08c639a95ae940808c97c5dfcdc13caf P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20240503 08:11:19.926081 22217 raft_consensus.cc:348] T 08c639a95ae940808c97c5dfcdc13caf P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:19.926203 22215 raft_consensus.cc:348] T 977a045f5f4b469f846aad8acf86b0b0 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:19.926410 22215 raft_consensus.cc:374] T 977a045f5f4b469f846aad8acf86b0b0 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.926579 22215 raft_consensus.cc:729] T 977a045f5f4b469f846aad8acf86b0b0 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.926982 22215 consensus_queue.cc:260] T 977a045f5f4b469f846aad8acf86b0b0 P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:19.927744 22217 raft_consensus.cc:374] T 08c639a95ae940808c97c5dfcdc13caf P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.928005 22217 raft_consensus.cc:729] T 08c639a95ae940808c97c5dfcdc13caf P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.928565 22217 consensus_queue.cc:260] T 08c639a95ae940808c97c5dfcdc13caf P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:19.928761 22215 ts_tablet_manager.cc:1398] T 977a045f5f4b469f846aad8acf86b0b0 P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:19.929262 22215 tablet_bootstrap.cc:492] T 08c639a95ae940808c97c5dfcdc13caf P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:19.929536 22217 ts_tablet_manager.cc:1398] T 08c639a95ae940808c97c5dfcdc13caf P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:11:19.931136 22215 tablet_bootstrap.cc:654] T 08c639a95ae940808c97c5dfcdc13caf P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.931483 22217 tablet_bootstrap.cc:492] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:19.931581 22220 tablet_bootstrap.cc:492] T 64a39d113638472580a5932bcc8531a3 P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:19.933610 22221 tablet_bootstrap.cc:492] T 64a39d113638472580a5932bcc8531a3 P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:19.934273 22217 tablet_bootstrap.cc:654] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.936026 22220 tablet_bootstrap.cc:654] T 64a39d113638472580a5932bcc8531a3 P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.936300 22221 tablet_bootstrap.cc:654] T 64a39d113638472580a5932bcc8531a3 P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.939941 22224 tablet_bootstrap.cc:492] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:19.942080 22224 tablet_bootstrap.cc:654] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.942353 22216 tablet_bootstrap.cc:492] T 977a045f5f4b469f846aad8acf86b0b0 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:19.942523 22220 tablet_bootstrap.cc:492] T 64a39d113638472580a5932bcc8531a3 P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:19.942677 22220 ts_tablet_manager.cc:1367] T 64a39d113638472580a5932bcc8531a3 P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 08:11:19.943347 22215 tablet_bootstrap.cc:492] T 08c639a95ae940808c97c5dfcdc13caf P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:19.943511 22215 ts_tablet_manager.cc:1367] T 08c639a95ae940808c97c5dfcdc13caf P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20240503 08:11:19.943497 22220 raft_consensus.cc:348] T 64a39d113638472580a5932bcc8531a3 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:19.943713 22220 raft_consensus.cc:374] T 64a39d113638472580a5932bcc8531a3 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.943812 22220 raft_consensus.cc:729] T 64a39d113638472580a5932bcc8531a3 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.944176 22216 tablet_bootstrap.cc:654] T 977a045f5f4b469f846aad8acf86b0b0 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.944052 22220 consensus_queue.cc:260] T 64a39d113638472580a5932bcc8531a3 P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:19.944221 22215 raft_consensus.cc:348] T 08c639a95ae940808c97c5dfcdc13caf P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:19.944485 22215 raft_consensus.cc:374] T 08c639a95ae940808c97c5dfcdc13caf P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.944604 22215 raft_consensus.cc:729] T 08c639a95ae940808c97c5dfcdc13caf P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.944664 22220 ts_tablet_manager.cc:1398] T 64a39d113638472580a5932bcc8531a3 P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:11:19.944931 22215 consensus_queue.cc:260] T 08c639a95ae940808c97c5dfcdc13caf P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:19.945732 22215 ts_tablet_manager.cc:1398] T 08c639a95ae940808c97c5dfcdc13caf P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:19.946103 22216 tablet_bootstrap.cc:492] T 977a045f5f4b469f846aad8acf86b0b0 P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:19.946274 22216 ts_tablet_manager.cc:1367] T 977a045f5f4b469f846aad8acf86b0b0 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:19.946818 22217 tablet_bootstrap.cc:492] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:19.947031 22217 ts_tablet_manager.cc:1367] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20240503 08:11:19.947131 22216 raft_consensus.cc:348] T 977a045f5f4b469f846aad8acf86b0b0 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:19.947391 22216 raft_consensus.cc:374] T 977a045f5f4b469f846aad8acf86b0b0 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.947535 22216 raft_consensus.cc:729] T 977a045f5f4b469f846aad8acf86b0b0 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.947798 22216 consensus_queue.cc:260] T 977a045f5f4b469f846aad8acf86b0b0 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:19.947840 22217 raft_consensus.cc:348] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:19.948050 22217 raft_consensus.cc:374] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.948199 22217 raft_consensus.cc:729] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.948491 22217 consensus_queue.cc:260] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:19.949040 22221 tablet_bootstrap.cc:492] T 64a39d113638472580a5932bcc8531a3 P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:19.949118 22216 ts_tablet_manager.cc:1398] T 977a045f5f4b469f846aad8acf86b0b0 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:19.949190 22221 ts_tablet_manager.cc:1367] T 64a39d113638472580a5932bcc8531a3 P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240503 08:11:19.949393 22216 tablet_bootstrap.cc:492] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:19.949903 22217 ts_tablet_manager.cc:1398] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:19.950055 22221 raft_consensus.cc:348] T 64a39d113638472580a5932bcc8531a3 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:19.950313 22221 raft_consensus.cc:374] T 64a39d113638472580a5932bcc8531a3 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.950413 22221 raft_consensus.cc:729] T 64a39d113638472580a5932bcc8531a3 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.950706 22221 consensus_queue.cc:260] T 64a39d113638472580a5932bcc8531a3 P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:19.950938 22216 tablet_bootstrap.cc:654] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.951429 22221 ts_tablet_manager.cc:1398] T 64a39d113638472580a5932bcc8531a3 P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:19.953442 21981 raft_consensus.cc:482] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:19.953649 21981 raft_consensus.cc:504] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:19.954526 21981 leader_election.cc:288] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:19.955303 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977a045f5f4b469f846aad8acf86b0b0" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" is_pre_election: true
I20240503 08:11:19.955579 21619 raft_consensus.cc:2454] T 977a045f5f4b469f846aad8acf86b0b0 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 0.
I20240503 08:11:19.956303 21292 leader_election.cc:302] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:19.957293 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977a045f5f4b469f846aad8acf86b0b0" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd" is_pre_election: true
I20240503 08:11:19.957604 21094 raft_consensus.cc:2454] T 977a045f5f4b469f846aad8acf86b0b0 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 0.
I20240503 08:11:19.958830 22216 tablet_bootstrap.cc:492] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:19.959023 22216 ts_tablet_manager.cc:1367] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240503 08:11:19.959501 22224 tablet_bootstrap.cc:492] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:19.959651 22224 ts_tablet_manager.cc:1367] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.008s
I20240503 08:11:19.959818 22216 raft_consensus.cc:348] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:19.960088 22216 raft_consensus.cc:374] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.960193 22216 raft_consensus.cc:729] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.960516 21981 raft_consensus.cc:2781] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:19.960465 22216 consensus_queue.cc:260] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:19.960621 21981 raft_consensus.cc:482] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:19.960583 22224 raft_consensus.cc:348] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:19.960731 21981 raft_consensus.cc:3037] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:19.960799 22224 raft_consensus.cc:374] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.960916 22224 raft_consensus.cc:729] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.961159 22216 ts_tablet_manager.cc:1398] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:19.961441 22224 consensus_queue.cc:260] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:19.962028 22224 ts_tablet_manager.cc:1398] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:19.963600 21981 raft_consensus.cc:504] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:19.967233 21981 leader_election.cc:288] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 election: Requested vote from peers 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:19.967716 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977a045f5f4b469f846aad8acf86b0b0" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd"
I20240503 08:11:19.967758 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977a045f5f4b469f846aad8acf86b0b0" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
I20240503 08:11:19.967949 21619 raft_consensus.cc:3037] T 977a045f5f4b469f846aad8acf86b0b0 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:19.967970 21094 raft_consensus.cc:3037] T 977a045f5f4b469f846aad8acf86b0b0 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:19.970865 21094 raft_consensus.cc:2454] T 977a045f5f4b469f846aad8acf86b0b0 P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 1.
I20240503 08:11:19.970885 21619 raft_consensus.cc:2454] T 977a045f5f4b469f846aad8acf86b0b0 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 1.
I20240503 08:11:19.971393 21292 leader_election.cc:302] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:19.971724 21981 raft_consensus.cc:2781] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:19.972173 21981 raft_consensus.cc:686] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 LEADER]: Becoming Leader. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Running, Role: LEADER
I20240503 08:11:19.972463 21981 consensus_queue.cc:237] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:19.976509 20948 catalog_manager.cc:5484] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131). New cstate: current_term: 1 leader_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:20.008508 22175 raft_consensus.cc:482] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:20.008752 22175 raft_consensus.cc:504] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:20.009454 22175 leader_election.cc:288] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:20.009965 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c639a95ae940808c97c5dfcdc13caf" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" is_pre_election: true
I20240503 08:11:20.010100 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c639a95ae940808c97c5dfcdc13caf" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd" is_pre_election: true
I20240503 08:11:20.010216 21356 raft_consensus.cc:2454] T 08c639a95ae940808c97c5dfcdc13caf P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 0.
I20240503 08:11:20.010341 21094 raft_consensus.cc:2454] T 08c639a95ae940808c97c5dfcdc13caf P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 0.
I20240503 08:11:20.010710 21552 leader_election.cc:302] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:20.011107 22175 raft_consensus.cc:2781] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:20.011199 22175 raft_consensus.cc:482] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:20.011327 22175 raft_consensus.cc:3037] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.013217 22175 raft_consensus.cc:504] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:20.013759 22175 leader_election.cc:288] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 election: Requested vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:20.014277 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c639a95ae940808c97c5dfcdc13caf" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd"
I20240503 08:11:20.014277 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c639a95ae940808c97c5dfcdc13caf" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5"
I20240503 08:11:20.014500 21356 raft_consensus.cc:3037] T 08c639a95ae940808c97c5dfcdc13caf P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.014500 21094 raft_consensus.cc:3037] T 08c639a95ae940808c97c5dfcdc13caf P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.017509 21094 raft_consensus.cc:2454] T 08c639a95ae940808c97c5dfcdc13caf P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 1.
I20240503 08:11:20.017547 21356 raft_consensus.cc:2454] T 08c639a95ae940808c97c5dfcdc13caf P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 1.
I20240503 08:11:20.017961 21553 leader_election.cc:302] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 18d9776f5d634ec2b167dc225659e4fd; no voters:
I20240503 08:11:20.018266 22175 raft_consensus.cc:2781] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:20.018436 22175 raft_consensus.cc:686] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 LEADER]: Becoming Leader. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Running, Role: LEADER
I20240503 08:11:20.018694 22175 consensus_queue.cc:237] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:20.021735 20948 catalog_manager.cc:5484] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133). New cstate: current_term: 1 leader_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:20.095248 22103 raft_consensus.cc:482] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:20.095474 22103 raft_consensus.cc:504] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:20.096222 22103 leader_election.cc:288] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:20.096578 21487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a39d113638472580a5932bcc8531a3" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842" is_pre_election: true
I20240503 08:11:20.096736 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a39d113638472580a5932bcc8531a3" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415" is_pre_election: true
I20240503 08:11:20.096900 21487 raft_consensus.cc:2454] T 64a39d113638472580a5932bcc8531a3 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 0.
I20240503 08:11:20.096997 21750 raft_consensus.cc:2454] T 64a39d113638472580a5932bcc8531a3 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 0.
I20240503 08:11:20.097328 21160 leader_election.cc:302] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 3e04594b1a5344d9bfefa4de05d70842; no voters:
I20240503 08:11:20.097604 22103 raft_consensus.cc:2781] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:20.097707 22103 raft_consensus.cc:482] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:20.097798 22103 raft_consensus.cc:3037] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.099632 22103 raft_consensus.cc:504] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:20.100162 22103 leader_election.cc:288] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 election: Requested vote from peers 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:20.100503 21487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a39d113638472580a5932bcc8531a3" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842"
I20240503 08:11:20.100620 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a39d113638472580a5932bcc8531a3" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415"
I20240503 08:11:20.100670 21487 raft_consensus.cc:3037] T 64a39d113638472580a5932bcc8531a3 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.100813 21750 raft_consensus.cc:3037] T 64a39d113638472580a5932bcc8531a3 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.102514 21487 raft_consensus.cc:2454] T 64a39d113638472580a5932bcc8531a3 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 1.
I20240503 08:11:20.102567 21750 raft_consensus.cc:2454] T 64a39d113638472580a5932bcc8531a3 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 1.
I20240503 08:11:20.102914 21160 leader_election.cc:302] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 3e04594b1a5344d9bfefa4de05d70842; no voters:
I20240503 08:11:20.103201 22103 raft_consensus.cc:2781] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:20.103343 22103 raft_consensus.cc:686] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 LEADER]: Becoming Leader. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Running, Role: LEADER
I20240503 08:11:20.103628 22103 consensus_queue.cc:237] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:20.106505 20948 catalog_manager.cc:5484] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130). New cstate: current_term: 1 leader_uuid: "3b6eea641d214c79988ec6ad8821a4bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:20.422109 21981 consensus_queue.cc:1035] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:20.428723 22237 consensus_queue.cc:1035] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:20.430323 22180 consensus_queue.cc:1035] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:20.439060 22175 consensus_queue.cc:1035] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:20.475739 22234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:20.476078 22234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:20.483650 22234 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 08:11:20.570647 22103 consensus_queue.cc:1035] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:20.584837 22103 consensus_queue.cc:1035] T 64a39d113638472580a5932bcc8531a3 P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 255.65 ms
time per row: 0.127825 ms
I20240503 08:11:20.897442 20948 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41046:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:20.906122 21205 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5a2cab7232499bb4aba2c6d8713723 (DEFAULT_TABLE table=rebalance_test_table_5 [id=796c2797aaf741618df73bcb288a17a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.906545 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d5a2cab7232499bb4aba2c6d8713723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.906441 21729 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5a2cab7232499bb4aba2c6d8713723 (DEFAULT_TABLE table=rebalance_test_table_5 [id=796c2797aaf741618df73bcb288a17a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.906899 21729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d5a2cab7232499bb4aba2c6d8713723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.907476 21467 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5a2cab7232499bb4aba2c6d8713723 (DEFAULT_TABLE table=rebalance_test_table_5 [id=796c2797aaf741618df73bcb288a17a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.907857 21467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d5a2cab7232499bb4aba2c6d8713723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.909247 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4d3741c904464dba15f0659614d1f8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=796c2797aaf741618df73bcb288a17a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.909626 21730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4d3741c904464dba15f0659614d1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.912050 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4d3741c904464dba15f0659614d1f8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=796c2797aaf741618df73bcb288a17a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.912433 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4d3741c904464dba15f0659614d1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.912541 21598 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe5f486eb394f12b698cb506e51ab3b (DEFAULT_TABLE table=rebalance_test_table_5 [id=796c2797aaf741618df73bcb288a17a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.912945 21598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe5f486eb394f12b698cb506e51ab3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.913244 21468 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4d3741c904464dba15f0659614d1f8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=796c2797aaf741618df73bcb288a17a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.913623 21468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4d3741c904464dba15f0659614d1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.918099 21073 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe5f486eb394f12b698cb506e51ab3b (DEFAULT_TABLE table=rebalance_test_table_5 [id=796c2797aaf741618df73bcb288a17a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.918516 21073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe5f486eb394f12b698cb506e51ab3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.921097 22297 tablet_bootstrap.cc:492] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:20.921872 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe5f486eb394f12b698cb506e51ab3b (DEFAULT_TABLE table=rebalance_test_table_5 [id=796c2797aaf741618df73bcb288a17a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.922272 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe5f486eb394f12b698cb506e51ab3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.922798 22297 tablet_bootstrap.cc:654] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.924060 22299 tablet_bootstrap.cc:492] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:20.925002 22298 tablet_bootstrap.cc:492] T 2b4d3741c904464dba15f0659614d1f8 P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:20.925756 22299 tablet_bootstrap.cc:654] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.926515 22298 tablet_bootstrap.cc:654] T 2b4d3741c904464dba15f0659614d1f8 P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.928645 22298 tablet_bootstrap.cc:492] T 2b4d3741c904464dba15f0659614d1f8 P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:20.928779 22298 ts_tablet_manager.cc:1367] T 2b4d3741c904464dba15f0659614d1f8 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:11:20.929476 22298 raft_consensus.cc:348] T 2b4d3741c904464dba15f0659614d1f8 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:20.929651 22298 raft_consensus.cc:374] T 2b4d3741c904464dba15f0659614d1f8 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.929759 22298 raft_consensus.cc:729] T 2b4d3741c904464dba15f0659614d1f8 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.929965 22298 consensus_queue.cc:260] T 2b4d3741c904464dba15f0659614d1f8 P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:20.930430 22298 ts_tablet_manager.cc:1398] T 2b4d3741c904464dba15f0659614d1f8 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:20.930683 22298 tablet_bootstrap.cc:492] T 4d5a2cab7232499bb4aba2c6d8713723 P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:20.932226 22298 tablet_bootstrap.cc:654] T 4d5a2cab7232499bb4aba2c6d8713723 P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.936816 22305 tablet_bootstrap.cc:492] T 5fe5f486eb394f12b698cb506e51ab3b P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:20.938474 22305 tablet_bootstrap.cc:654] T 5fe5f486eb394f12b698cb506e51ab3b P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.939958 22298 tablet_bootstrap.cc:492] T 4d5a2cab7232499bb4aba2c6d8713723 P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:20.940114 22298 ts_tablet_manager.cc:1367] T 4d5a2cab7232499bb4aba2c6d8713723 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20240503 08:11:20.940687 22298 raft_consensus.cc:348] T 4d5a2cab7232499bb4aba2c6d8713723 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:20.940892 22298 raft_consensus.cc:374] T 4d5a2cab7232499bb4aba2c6d8713723 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.940979 22298 raft_consensus.cc:729] T 4d5a2cab7232499bb4aba2c6d8713723 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.941223 22298 consensus_queue.cc:260] T 4d5a2cab7232499bb4aba2c6d8713723 P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:20.941826 22298 ts_tablet_manager.cc:1398] T 4d5a2cab7232499bb4aba2c6d8713723 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:20.942902 22305 tablet_bootstrap.cc:492] T 5fe5f486eb394f12b698cb506e51ab3b P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:20.943068 22305 ts_tablet_manager.cc:1367] T 5fe5f486eb394f12b698cb506e51ab3b P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:11:20.943923 22305 raft_consensus.cc:348] T 5fe5f486eb394f12b698cb506e51ab3b P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:20.944157 22305 raft_consensus.cc:374] T 5fe5f486eb394f12b698cb506e51ab3b P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.944283 22305 raft_consensus.cc:729] T 5fe5f486eb394f12b698cb506e51ab3b P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.944556 22305 consensus_queue.cc:260] T 5fe5f486eb394f12b698cb506e51ab3b P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:20.945911 22305 ts_tablet_manager.cc:1398] T 5fe5f486eb394f12b698cb506e51ab3b P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:20.946808 22307 tablet_bootstrap.cc:492] T 5fe5f486eb394f12b698cb506e51ab3b P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:20.947324 22303 tablet_bootstrap.cc:492] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:20.949118 22303 tablet_bootstrap.cc:654] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.951112 22307 tablet_bootstrap.cc:654] T 5fe5f486eb394f12b698cb506e51ab3b P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.953879 22297 tablet_bootstrap.cc:492] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:20.954130 22297 ts_tablet_manager.cc:1367] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.007s
I20240503 08:11:20.955104 22297 raft_consensus.cc:348] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:20.955310 22297 raft_consensus.cc:374] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.955431 22297 raft_consensus.cc:729] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.955586 22303 tablet_bootstrap.cc:492] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:20.955735 22303 ts_tablet_manager.cc:1367] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240503 08:11:20.955775 22299 tablet_bootstrap.cc:492] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:20.955714 22297 consensus_queue.cc:260] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:20.955912 22299 ts_tablet_manager.cc:1367] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.015s
I20240503 08:11:20.956264 22297 ts_tablet_manager.cc:1398] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:20.956569 22297 tablet_bootstrap.cc:492] T 2b4d3741c904464dba15f0659614d1f8 P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:20.956786 22303 raft_consensus.cc:348] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:20.957003 22303 raft_consensus.cc:374] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.957103 22303 raft_consensus.cc:729] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.957377 22299 raft_consensus.cc:348] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:20.957461 22303 consensus_queue.cc:260] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:20.957628 22299 raft_consensus.cc:374] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.957765 22299 raft_consensus.cc:729] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.958043 22299 consensus_queue.cc:260] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:20.958495 22297 tablet_bootstrap.cc:654] T 2b4d3741c904464dba15f0659614d1f8 P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.958559 22140 raft_consensus.cc:482] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:20.958693 22140 raft_consensus.cc:504] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:20.958937 22299 ts_tablet_manager.cc:1398] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:11:20.959370 22140 leader_election.cc:288] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:20.959782 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4d3741c904464dba15f0659614d1f8" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf" is_pre_election: true
I20240503 08:11:20.960022 21226 raft_consensus.cc:2454] T 2b4d3741c904464dba15f0659614d1f8 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 0.
I20240503 08:11:20.960012 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4d3741c904464dba15f0659614d1f8" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415" is_pre_election: true
I20240503 08:11:20.960554 21423 leader_election.cc:302] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 3e04594b1a5344d9bfefa4de05d70842; no voters:
W20240503 08:11:20.960763 21424 leader_election.cc:341] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6503743925434401a74aae303628e415 (127.20.107.134:42791): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:20.960927 22140 raft_consensus.cc:2781] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:20.961032 22140 raft_consensus.cc:482] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:20.961112 22140 raft_consensus.cc:3037] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.961586 22303 ts_tablet_manager.cc:1398] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:11:20.961963 22303 tablet_bootstrap.cc:492] T 4d5a2cab7232499bb4aba2c6d8713723 P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:20.963892 22140 raft_consensus.cc:504] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:20.964816 22303 tablet_bootstrap.cc:654] T 4d5a2cab7232499bb4aba2c6d8713723 P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.964977 22307 tablet_bootstrap.cc:492] T 5fe5f486eb394f12b698cb506e51ab3b P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:20.964944 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4d3741c904464dba15f0659614d1f8" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf"
I20240503 08:11:20.965160 22307 ts_tablet_manager.cc:1367] T 5fe5f486eb394f12b698cb506e51ab3b P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240503 08:11:20.965181 21226 raft_consensus.cc:3037] T 2b4d3741c904464dba15f0659614d1f8 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.965967 22307 raft_consensus.cc:348] T 5fe5f486eb394f12b698cb506e51ab3b P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:20.966171 22307 raft_consensus.cc:374] T 5fe5f486eb394f12b698cb506e51ab3b P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.966252 22307 raft_consensus.cc:729] T 5fe5f486eb394f12b698cb506e51ab3b P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.966528 22297 tablet_bootstrap.cc:492] T 2b4d3741c904464dba15f0659614d1f8 P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:20.966468 22307 consensus_queue.cc:260] T 5fe5f486eb394f12b698cb506e51ab3b P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:20.966701 22297 ts_tablet_manager.cc:1367] T 2b4d3741c904464dba15f0659614d1f8 P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240503 08:11:20.967470 22297 raft_consensus.cc:348] T 2b4d3741c904464dba15f0659614d1f8 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:20.967680 22297 raft_consensus.cc:374] T 2b4d3741c904464dba15f0659614d1f8 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.967756 22297 raft_consensus.cc:729] T 2b4d3741c904464dba15f0659614d1f8 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.967897 21226 raft_consensus.cc:2454] T 2b4d3741c904464dba15f0659614d1f8 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:20.968029 22297 consensus_queue.cc:260] T 2b4d3741c904464dba15f0659614d1f8 P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:20.968516 21423 leader_election.cc:302] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 3e04594b1a5344d9bfefa4de05d70842; no voters:
I20240503 08:11:20.968654 22297 ts_tablet_manager.cc:1398] T 2b4d3741c904464dba15f0659614d1f8 P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:20.968775 22307 ts_tablet_manager.cc:1398] T 5fe5f486eb394f12b698cb506e51ab3b P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:20.969209 22312 raft_consensus.cc:2781] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:20.969842 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4d3741c904464dba15f0659614d1f8" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415"
I20240503 08:11:20.970018 21750 raft_consensus.cc:3037] T 2b4d3741c904464dba15f0659614d1f8 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.970578 22303 tablet_bootstrap.cc:492] T 4d5a2cab7232499bb4aba2c6d8713723 P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:20.970746 22303 ts_tablet_manager.cc:1367] T 4d5a2cab7232499bb4aba2c6d8713723 P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240503 08:11:20.971585 22303 raft_consensus.cc:348] T 4d5a2cab7232499bb4aba2c6d8713723 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:20.971797 22303 raft_consensus.cc:374] T 4d5a2cab7232499bb4aba2c6d8713723 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.971879 22303 raft_consensus.cc:729] T 4d5a2cab7232499bb4aba2c6d8713723 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.972040 21750 raft_consensus.cc:2454] T 2b4d3741c904464dba15f0659614d1f8 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:20.972119 22303 consensus_queue.cc:260] T 4d5a2cab7232499bb4aba2c6d8713723 P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:20.972306 22140 leader_election.cc:288] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:20.973158 22303 ts_tablet_manager.cc:1398] T 4d5a2cab7232499bb4aba2c6d8713723 P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:11:20.973358 22312 raft_consensus.cc:686] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 LEADER]: Becoming Leader. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Running, Role: LEADER
I20240503 08:11:20.973667 22312 consensus_queue.cc:237] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:20.976589 20948 catalog_manager.cc:5484] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132). New cstate: current_term: 1 leader_uuid: "3e04594b1a5344d9bfefa4de05d70842" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:21.084518 22180 raft_consensus.cc:482] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:21.084753 22180 raft_consensus.cc:504] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:21.085477 22180 leader_election.cc:288] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:21.086032 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe5f486eb394f12b698cb506e51ab3b" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd" is_pre_election: true
I20240503 08:11:21.086035 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe5f486eb394f12b698cb506e51ab3b" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" is_pre_election: true
I20240503 08:11:21.086267 21356 raft_consensus.cc:2454] T 5fe5f486eb394f12b698cb506e51ab3b P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 0.
I20240503 08:11:21.086257 21094 raft_consensus.cc:2454] T 5fe5f486eb394f12b698cb506e51ab3b P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 0.
I20240503 08:11:21.086714 21552 leader_election.cc:302] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:21.087025 22180 raft_consensus.cc:2781] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:21.087128 22180 raft_consensus.cc:482] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:21.087229 22180 raft_consensus.cc:3037] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:21.089804 22180 raft_consensus.cc:504] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:21.090396 22180 leader_election.cc:288] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 election: Requested vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:21.090736 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe5f486eb394f12b698cb506e51ab3b" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5"
I20240503 08:11:21.090849 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe5f486eb394f12b698cb506e51ab3b" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd"
I20240503 08:11:21.090902 21839 raft_consensus.cc:482] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:21.090953 21356 raft_consensus.cc:3037] T 5fe5f486eb394f12b698cb506e51ab3b P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:21.091063 21094 raft_consensus.cc:3037] T 5fe5f486eb394f12b698cb506e51ab3b P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:21.091073 21839 raft_consensus.cc:504] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:21.091722 21839 leader_election.cc:288] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603)
I20240503 08:11:21.092149 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5a2cab7232499bb4aba2c6d8713723" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf" is_pre_election: true
I20240503 08:11:21.092384 21226 raft_consensus.cc:2454] T 4d5a2cab7232499bb4aba2c6d8713723 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 0.
I20240503 08:11:21.092396 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5a2cab7232499bb4aba2c6d8713723" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842" is_pre_election: true
I20240503 08:11:21.092593 21488 raft_consensus.cc:2454] T 4d5a2cab7232499bb4aba2c6d8713723 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 0.
I20240503 08:11:21.092830 21685 leader_election.cc:302] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:21.093353 21839 raft_consensus.cc:2781] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:21.093443 21839 raft_consensus.cc:482] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:21.093569 21839 raft_consensus.cc:3037] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:21.093816 21094 raft_consensus.cc:2454] T 5fe5f486eb394f12b698cb506e51ab3b P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 1.
I20240503 08:11:21.093816 21356 raft_consensus.cc:2454] T 5fe5f486eb394f12b698cb506e51ab3b P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 1.
I20240503 08:11:21.094190 21552 leader_election.cc:302] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:21.094446 22180 raft_consensus.cc:2781] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:21.094661 22180 raft_consensus.cc:686] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 LEADER]: Becoming Leader. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Running, Role: LEADER
I20240503 08:11:21.094923 22180 consensus_queue.cc:237] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:21.095979 21839 raft_consensus.cc:504] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:21.097054 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5a2cab7232499bb4aba2c6d8713723" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf"
I20240503 08:11:21.097210 21839 leader_election.cc:288] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603)
I20240503 08:11:21.097275 21226 raft_consensus.cc:3037] T 4d5a2cab7232499bb4aba2c6d8713723 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:21.097574 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5a2cab7232499bb4aba2c6d8713723" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842"
I20240503 08:11:21.097759 21488 raft_consensus.cc:3037] T 4d5a2cab7232499bb4aba2c6d8713723 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:21.098551 20948 catalog_manager.cc:5484] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133). New cstate: current_term: 1 leader_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:21.100028 21226 raft_consensus.cc:2454] T 4d5a2cab7232499bb4aba2c6d8713723 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 1.
I20240503 08:11:21.100440 21488 raft_consensus.cc:2454] T 4d5a2cab7232499bb4aba2c6d8713723 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 1.
I20240503 08:11:21.100526 21685 leader_election.cc:302] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:21.100802 21839 raft_consensus.cc:2781] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:21.101331 21839 raft_consensus.cc:686] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [term 1 LEADER]: Becoming Leader. State: Replica: 6503743925434401a74aae303628e415, State: Running, Role: LEADER
I20240503 08:11:21.101620 21839 consensus_queue.cc:237] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:21.104383 20948 catalog_manager.cc:5484] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6503743925434401a74aae303628e415 (127.20.107.134). New cstate: current_term: 1 leader_uuid: "6503743925434401a74aae303628e415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:21.474349 22312 consensus_queue.cc:1035] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:21.487437 22312 consensus_queue.cc:1035] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:21.490855 22180 consensus_queue.cc:1035] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:21.511485 22175 consensus_queue.cc:1035] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:21.515959 22319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:21.516264 22319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:21.525506 22319 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 08:11:21.621951 21839 consensus_queue.cc:1035] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:21.656168 22359 consensus_queue.cc:1035] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240503 08:11:21.853865 21029 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.133:43423, user_credentials={real_user=slave}} blocked reactor thread for 49642.2us
INSERT report
rows total: 2000
time total: 318.615 ms
time per row: 0.159307 ms
I20240503 08:11:21.991207 20948 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41046:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:21.999929 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 87221edea55a4bcc85e1a0d9bf8f0c55 (DEFAULT_TABLE table=rebalance_test_table_6 [id=8c8961bb28424b93abac47d973c080c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:22.000186 21073 tablet_service.cc:1467] Processing CreateTablet for tablet 87221edea55a4bcc85e1a0d9bf8f0c55 (DEFAULT_TABLE table=rebalance_test_table_6 [id=8c8961bb28424b93abac47d973c080c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:22.000420 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87221edea55a4bcc85e1a0d9bf8f0c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:22.000568 21073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87221edea55a4bcc85e1a0d9bf8f0c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:22.005326 21598 tablet_service.cc:1467] Processing CreateTablet for tablet 87221edea55a4bcc85e1a0d9bf8f0c55 (DEFAULT_TABLE table=rebalance_test_table_6 [id=8c8961bb28424b93abac47d973c080c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:22.005792 21598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87221edea55a4bcc85e1a0d9bf8f0c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:22.005725 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 0c52927816c44a9a87a5fa95ada7801a (DEFAULT_TABLE table=rebalance_test_table_6 [id=8c8961bb28424b93abac47d973c080c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:22.005862 21729 tablet_service.cc:1467] Processing CreateTablet for tablet 0c52927816c44a9a87a5fa95ada7801a (DEFAULT_TABLE table=rebalance_test_table_6 [id=8c8961bb28424b93abac47d973c080c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:22.006119 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c52927816c44a9a87a5fa95ada7801a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:22.006285 21729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c52927816c44a9a87a5fa95ada7801a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:22.006258 21599 tablet_service.cc:1467] Processing CreateTablet for tablet 2b12dd2cc18748d2a3251f6ad7e6e482 (DEFAULT_TABLE table=rebalance_test_table_6 [id=8c8961bb28424b93abac47d973c080c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:22.006613 21599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b12dd2cc18748d2a3251f6ad7e6e482. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:22.009486 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 2b12dd2cc18748d2a3251f6ad7e6e482 (DEFAULT_TABLE table=rebalance_test_table_6 [id=8c8961bb28424b93abac47d973c080c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:22.009852 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b12dd2cc18748d2a3251f6ad7e6e482. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:22.011982 21468 tablet_service.cc:1467] Processing CreateTablet for tablet 0c52927816c44a9a87a5fa95ada7801a (DEFAULT_TABLE table=rebalance_test_table_6 [id=8c8961bb28424b93abac47d973c080c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:22.012369 21468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c52927816c44a9a87a5fa95ada7801a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:22.013644 21074 tablet_service.cc:1467] Processing CreateTablet for tablet 2b12dd2cc18748d2a3251f6ad7e6e482 (DEFAULT_TABLE table=rebalance_test_table_6 [id=8c8961bb28424b93abac47d973c080c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:22.014046 21074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b12dd2cc18748d2a3251f6ad7e6e482. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:22.030571 22394 tablet_bootstrap.cc:492] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:22.030781 22392 tablet_bootstrap.cc:492] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:22.031607 22393 tablet_bootstrap.cc:492] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:22.032219 22395 tablet_bootstrap.cc:492] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:22.032471 22392 tablet_bootstrap.cc:654] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:22.032471 22394 tablet_bootstrap.cc:654] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:22.033591 22396 tablet_bootstrap.cc:492] T 0c52927816c44a9a87a5fa95ada7801a P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:22.035056 22397 tablet_bootstrap.cc:492] T 0c52927816c44a9a87a5fa95ada7801a P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:22.035885 22393 tablet_bootstrap.cc:654] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:22.036911 22396 tablet_bootstrap.cc:654] T 0c52927816c44a9a87a5fa95ada7801a P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:22.037043 22397 tablet_bootstrap.cc:654] T 0c52927816c44a9a87a5fa95ada7801a P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:22.037334 22395 tablet_bootstrap.cc:654] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:22.038807 22392 tablet_bootstrap.cc:492] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:22.038807 22394 tablet_bootstrap.cc:492] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:22.038960 22392 ts_tablet_manager.cc:1367] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 08:11:22.038960 22394 ts_tablet_manager.cc:1367] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240503 08:11:22.039753 22392 raft_consensus.cc:348] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:22.039753 22394 raft_consensus.cc:348] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:22.039973 22394 raft_consensus.cc:374] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:22.040061 22392 raft_consensus.cc:374] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:22.040109 22394 raft_consensus.cc:729] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:22.040184 22392 raft_consensus.cc:729] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:22.040397 22394 consensus_queue.cc:260] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:22.040441 22392 consensus_queue.cc:260] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:22.041044 22392 ts_tablet_manager.cc:1398] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:22.041133 22394 ts_tablet_manager.cc:1398] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:22.042063 22396 tablet_bootstrap.cc:492] T 0c52927816c44a9a87a5fa95ada7801a P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:22.042237 22396 ts_tablet_manager.cc:1367] T 0c52927816c44a9a87a5fa95ada7801a P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 08:11:22.042600 22397 tablet_bootstrap.cc:492] T 0c52927816c44a9a87a5fa95ada7801a P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:22.042765 22397 ts_tablet_manager.cc:1367] T 0c52927816c44a9a87a5fa95ada7801a P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240503 08:11:22.043166 22396 raft_consensus.cc:348] T 0c52927816c44a9a87a5fa95ada7801a P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:22.043435 22396 raft_consensus.cc:374] T 0c52927816c44a9a87a5fa95ada7801a P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:22.043545 22396 raft_consensus.cc:729] T 0c52927816c44a9a87a5fa95ada7801a P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:22.043601 22397 raft_consensus.cc:348] T 0c52927816c44a9a87a5fa95ada7801a P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:22.043834 22397 raft_consensus.cc:374] T 0c52927816c44a9a87a5fa95ada7801a P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:22.043779 22396 consensus_queue.cc:260] T 0c52927816c44a9a87a5fa95ada7801a P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:22.043962 22397 raft_consensus.cc:729] T 0c52927816c44a9a87a5fa95ada7801a P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:22.044395 22396 ts_tablet_manager.cc:1398] T 0c52927816c44a9a87a5fa95ada7801a P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:22.044267 22397 consensus_queue.cc:260] T 0c52927816c44a9a87a5fa95ada7801a P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:22.045080 22397 ts_tablet_manager.cc:1398] T 0c52927816c44a9a87a5fa95ada7801a P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:22.047555 22394 tablet_bootstrap.cc:492] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:22.049281 22394 tablet_bootstrap.cc:654] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:22.052246 22393 tablet_bootstrap.cc:492] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:22.052392 22393 ts_tablet_manager.cc:1367] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20240503 08:11:22.053045 22394 tablet_bootstrap.cc:492] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:22.053040 22393 raft_consensus.cc:348] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:22.053221 22394 ts_tablet_manager.cc:1367] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 08:11:22.053304 22393 raft_consensus.cc:374] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:22.053422 22393 raft_consensus.cc:729] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:22.053663 22393 consensus_queue.cc:260] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:22.053860 22395 tablet_bootstrap.cc:492] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:22.054076 22395 ts_tablet_manager.cc:1367] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.000s
I20240503 08:11:22.054018 22394 raft_consensus.cc:348] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:22.054304 22394 raft_consensus.cc:374] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:22.054333 22393 ts_tablet_manager.cc:1398] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:22.054421 22394 raft_consensus.cc:729] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:22.054677 22393 tablet_bootstrap.cc:492] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:22.054731 22394 consensus_queue.cc:260] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:22.056010 22394 ts_tablet_manager.cc:1398] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:22.056232 22393 tablet_bootstrap.cc:654] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:22.056675 22395 raft_consensus.cc:348] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:22.057003 22395 raft_consensus.cc:374] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:22.057194 22395 raft_consensus.cc:729] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:22.057688 22395 consensus_queue.cc:260] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:22.058287 22395 ts_tablet_manager.cc:1398] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:11:22.060765 22395 tablet_bootstrap.cc:492] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:22.062186 22395 tablet_bootstrap.cc:654] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:22.064220 22395 tablet_bootstrap.cc:492] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:22.064240 22393 tablet_bootstrap.cc:492] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:22.064378 22395 ts_tablet_manager.cc:1367] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:22.064391 22393 ts_tablet_manager.cc:1367] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240503 08:11:22.065006 22395 raft_consensus.cc:348] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:22.065066 22393 raft_consensus.cc:348] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:22.065178 22395 raft_consensus.cc:374] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:22.065258 22395 raft_consensus.cc:729] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:22.065268 22393 raft_consensus.cc:374] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:22.065333 22393 raft_consensus.cc:729] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:22.065490 22395 consensus_queue.cc:260] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:22.065580 22393 consensus_queue.cc:260] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:22.066134 22393 ts_tablet_manager.cc:1398] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:22.066174 22395 ts_tablet_manager.cc:1398] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:22.107396 22268 raft_consensus.cc:482] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:22.107569 22268 raft_consensus.cc:504] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:22.108139 22268 leader_election.cc:288] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6503743925434401a74aae303628e415 (127.20.107.134:42791), 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603)
I20240503 08:11:22.108495 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c52927816c44a9a87a5fa95ada7801a" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415" is_pre_election: true
I20240503 08:11:22.108733 21750 raft_consensus.cc:2454] T 0c52927816c44a9a87a5fa95ada7801a P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 0.
I20240503 08:11:22.108665 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c52927816c44a9a87a5fa95ada7801a" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842" is_pre_election: true
I20240503 08:11:22.108942 21488 raft_consensus.cc:2454] T 0c52927816c44a9a87a5fa95ada7801a P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 0.
I20240503 08:11:22.109210 21162 leader_election.cc:302] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:22.109468 22268 raft_consensus.cc:2781] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:22.109572 22268 raft_consensus.cc:482] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:22.109659 22268 raft_consensus.cc:3037] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:22.111518 22268 raft_consensus.cc:504] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:22.112044 22268 leader_election.cc:288] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 election: Requested vote from peers 6503743925434401a74aae303628e415 (127.20.107.134:42791), 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603)
I20240503 08:11:22.112434 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c52927816c44a9a87a5fa95ada7801a" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415"
I20240503 08:11:22.112588 21750 raft_consensus.cc:3037] T 0c52927816c44a9a87a5fa95ada7801a P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:22.112525 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c52927816c44a9a87a5fa95ada7801a" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842"
I20240503 08:11:22.112744 21488 raft_consensus.cc:3037] T 0c52927816c44a9a87a5fa95ada7801a P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:22.114517 21750 raft_consensus.cc:2454] T 0c52927816c44a9a87a5fa95ada7801a P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 1.
I20240503 08:11:22.114775 21488 raft_consensus.cc:2454] T 0c52927816c44a9a87a5fa95ada7801a P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 1.
I20240503 08:11:22.114902 21162 leader_election.cc:302] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:22.115224 22268 raft_consensus.cc:2781] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:22.115638 22268 raft_consensus.cc:686] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [term 1 LEADER]: Becoming Leader. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Running, Role: LEADER
I20240503 08:11:22.115955 22268 consensus_queue.cc:237] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:22.118707 20952 catalog_manager.cc:5484] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130). New cstate: current_term: 1 leader_uuid: "3b6eea641d214c79988ec6ad8821a4bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:22.130549 22060 raft_consensus.cc:482] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:22.130751 22060 raft_consensus.cc:504] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:22.131436 22060 leader_election.cc:288] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:22.132000 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b12dd2cc18748d2a3251f6ad7e6e482" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" is_pre_election: true
I20240503 08:11:22.132000 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b12dd2cc18748d2a3251f6ad7e6e482" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" is_pre_election: true
I20240503 08:11:22.132243 21356 raft_consensus.cc:2454] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 0.
I20240503 08:11:22.132263 21619 raft_consensus.cc:2454] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 0.
I20240503 08:11:22.132704 21029 leader_election.cc:302] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 18d9776f5d634ec2b167dc225659e4fd; no voters:
I20240503 08:11:22.132993 22060 raft_consensus.cc:2781] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:22.133109 22060 raft_consensus.cc:482] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:22.133204 22060 raft_consensus.cc:3037] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:22.135134 22060 raft_consensus.cc:504] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:22.135715 22060 leader_election.cc:288] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 election: Requested vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:22.136169 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b12dd2cc18748d2a3251f6ad7e6e482" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
I20240503 08:11:22.136183 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b12dd2cc18748d2a3251f6ad7e6e482" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5"
I20240503 08:11:22.136376 21356 raft_consensus.cc:3037] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:22.136368 21619 raft_consensus.cc:3037] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:22.138265 21619 raft_consensus.cc:2454] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 1.
I20240503 08:11:22.138298 21356 raft_consensus.cc:2454] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 1.
I20240503 08:11:22.138665 21029 leader_election.cc:302] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 18d9776f5d634ec2b167dc225659e4fd; no voters:
I20240503 08:11:22.138926 22060 raft_consensus.cc:2781] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:22.139067 22060 raft_consensus.cc:686] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [term 1 LEADER]: Becoming Leader. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Running, Role: LEADER
I20240503 08:11:22.139362 22060 consensus_queue.cc:237] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:22.142110 20952 catalog_manager.cc:5484] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129). New cstate: current_term: 1 leader_uuid: "18d9776f5d634ec2b167dc225659e4fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:22.299630 22060 raft_consensus.cc:482] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:22.299850 22060 raft_consensus.cc:504] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:22.300494 22060 leader_election.cc:288] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:22.300938 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87221edea55a4bcc85e1a0d9bf8f0c55" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" is_pre_election: true
I20240503 08:11:22.301021 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87221edea55a4bcc85e1a0d9bf8f0c55" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" is_pre_election: true
I20240503 08:11:22.301206 21356 raft_consensus.cc:2454] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 0.
I20240503 08:11:22.301265 21619 raft_consensus.cc:2454] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 0.
I20240503 08:11:22.301698 21027 leader_election.cc:302] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d9776f5d634ec2b167dc225659e4fd, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:22.301975 22060 raft_consensus.cc:2781] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:22.302071 22060 raft_consensus.cc:482] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:22.302179 22060 raft_consensus.cc:3037] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:22.304070 22060 raft_consensus.cc:504] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:22.304761 22060 leader_election.cc:288] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 election: Requested vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:22.305073 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87221edea55a4bcc85e1a0d9bf8f0c55" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
I20240503 08:11:22.305147 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87221edea55a4bcc85e1a0d9bf8f0c55" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5"
I20240503 08:11:22.305330 21356 raft_consensus.cc:3037] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:22.305328 21619 raft_consensus.cc:3037] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:22.307154 21356 raft_consensus.cc:2454] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 1.
I20240503 08:11:22.307540 21027 leader_election.cc:302] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d9776f5d634ec2b167dc225659e4fd, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:22.307776 22060 raft_consensus.cc:2781] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:22.307941 22060 raft_consensus.cc:686] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [term 1 LEADER]: Becoming Leader. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Running, Role: LEADER
I20240503 08:11:22.308173 21619 raft_consensus.cc:2454] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 1.
I20240503 08:11:22.308223 22060 consensus_queue.cc:237] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:22.311256 20952 catalog_manager.cc:5484] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129). New cstate: current_term: 1 leader_uuid: "18d9776f5d634ec2b167dc225659e4fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:22.509361 22407 consensus_queue.cc:1035] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:22.519937 22268 consensus_queue.cc:1035] T 0c52927816c44a9a87a5fa95ada7801a P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:22.535949 22060 consensus_queue.cc:1035] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:22.542887 22060 consensus_queue.cc:1035] T 2b12dd2cc18748d2a3251f6ad7e6e482 P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:22.733840 22411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:22.734149 22411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:22.742120 22411 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:11:22.765179 22383 consensus_queue.cc:1035] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:22.770006 22060 consensus_queue.cc:1035] T 87221edea55a4bcc85e1a0d9bf8f0c55 P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
INSERT report
rows total: 2000
time total: 258.255 ms
time per row: 0.129127 ms
I20240503 08:11:23.148382 20947 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41046:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:23.156479 21468 tablet_service.cc:1467] Processing CreateTablet for tablet 28305e23ecff40e9bce63ccf908cecb5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d52787a296241e68b8b3149c77a5ff9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:23.156970 21468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28305e23ecff40e9bce63ccf908cecb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:23.156905 21729 tablet_service.cc:1467] Processing CreateTablet for tablet 28305e23ecff40e9bce63ccf908cecb5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d52787a296241e68b8b3149c77a5ff9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:23.157323 21729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28305e23ecff40e9bce63ccf908cecb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:23.157238 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 28305e23ecff40e9bce63ccf908cecb5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d52787a296241e68b8b3149c77a5ff9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:23.157621 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28305e23ecff40e9bce63ccf908cecb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:23.157783 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac07d68002548958c4eeede92fcd4be (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d52787a296241e68b8b3149c77a5ff9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:23.158167 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ac07d68002548958c4eeede92fcd4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:23.161597 21205 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0614c8c6ce4539925785c065829ad2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d52787a296241e68b8b3149c77a5ff9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:23.161993 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0614c8c6ce4539925785c065829ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:23.165622 21467 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0614c8c6ce4539925785c065829ad2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d52787a296241e68b8b3149c77a5ff9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:23.166013 21467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0614c8c6ce4539925785c065829ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:23.166129 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0614c8c6ce4539925785c065829ad2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d52787a296241e68b8b3149c77a5ff9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:23.166502 21730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0614c8c6ce4539925785c065829ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:23.171980 22478 tablet_bootstrap.cc:492] T 28305e23ecff40e9bce63ccf908cecb5 P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:23.172063 21598 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac07d68002548958c4eeede92fcd4be (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d52787a296241e68b8b3149c77a5ff9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:23.172487 21598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ac07d68002548958c4eeede92fcd4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:23.173931 22478 tablet_bootstrap.cc:654] T 28305e23ecff40e9bce63ccf908cecb5 P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:23.176054 21073 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac07d68002548958c4eeede92fcd4be (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d52787a296241e68b8b3149c77a5ff9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:23.176729 21073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ac07d68002548958c4eeede92fcd4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:23.178714 22479 tablet_bootstrap.cc:492] T 28305e23ecff40e9bce63ccf908cecb5 P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:23.180982 22479 tablet_bootstrap.cc:654] T 28305e23ecff40e9bce63ccf908cecb5 P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:23.192164 22484 tablet_bootstrap.cc:492] T 5ac07d68002548958c4eeede92fcd4be P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:23.193516 22483 tablet_bootstrap.cc:492] T 5ac07d68002548958c4eeede92fcd4be P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:23.194465 22484 tablet_bootstrap.cc:654] T 5ac07d68002548958c4eeede92fcd4be P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:23.194849 22482 tablet_bootstrap.cc:492] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:23.197572 22479 tablet_bootstrap.cc:492] T 28305e23ecff40e9bce63ccf908cecb5 P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:23.197808 22477 tablet_bootstrap.cc:492] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:23.198199 22482 tablet_bootstrap.cc:654] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:23.198462 22478 tablet_bootstrap.cc:492] T 28305e23ecff40e9bce63ccf908cecb5 P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:23.198626 22478 ts_tablet_manager.cc:1367] T 28305e23ecff40e9bce63ccf908cecb5 P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.013s
I20240503 08:11:23.199335 22477 tablet_bootstrap.cc:654] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:23.199462 22478 raft_consensus.cc:348] T 28305e23ecff40e9bce63ccf908cecb5 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.199632 22483 tablet_bootstrap.cc:654] T 5ac07d68002548958c4eeede92fcd4be P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:23.199704 22478 raft_consensus.cc:374] T 28305e23ecff40e9bce63ccf908cecb5 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:23.199831 22478 raft_consensus.cc:729] T 28305e23ecff40e9bce63ccf908cecb5 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:23.200098 22478 consensus_queue.cc:260] T 28305e23ecff40e9bce63ccf908cecb5 P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.200261 22479 ts_tablet_manager.cc:1367] T 28305e23ecff40e9bce63ccf908cecb5 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.007s
I20240503 08:11:23.200927 22478 ts_tablet_manager.cc:1398] T 28305e23ecff40e9bce63ccf908cecb5 P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:23.201063 22479 raft_consensus.cc:348] T 28305e23ecff40e9bce63ccf908cecb5 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.201303 22478 tablet_bootstrap.cc:492] T 7f0614c8c6ce4539925785c065829ad2 P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:23.201282 22479 raft_consensus.cc:374] T 28305e23ecff40e9bce63ccf908cecb5 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:23.201710 22479 raft_consensus.cc:729] T 28305e23ecff40e9bce63ccf908cecb5 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:23.201988 22479 consensus_queue.cc:260] T 28305e23ecff40e9bce63ccf908cecb5 P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.202584 22479 ts_tablet_manager.cc:1398] T 28305e23ecff40e9bce63ccf908cecb5 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:23.203843 22478 tablet_bootstrap.cc:654] T 7f0614c8c6ce4539925785c065829ad2 P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:23.204207 22482 tablet_bootstrap.cc:492] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:23.205056 22479 tablet_bootstrap.cc:492] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:23.206472 22482 ts_tablet_manager.cc:1367] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 08:11:23.207115 22482 raft_consensus.cc:348] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:23.207384 22482 raft_consensus.cc:374] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:23.207541 22482 raft_consensus.cc:729] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:23.207806 22482 consensus_queue.cc:260] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:23.208590 22477 tablet_bootstrap.cc:492] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:23.208693 22479 tablet_bootstrap.cc:654] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:23.209630 22484 tablet_bootstrap.cc:492] T 5ac07d68002548958c4eeede92fcd4be P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:23.209779 22484 ts_tablet_manager.cc:1367] T 5ac07d68002548958c4eeede92fcd4be P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.003s
I20240503 08:11:23.210247 22482 ts_tablet_manager.cc:1398] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:11:23.210567 22477 ts_tablet_manager.cc:1367] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 08:11:23.210583 22484 raft_consensus.cc:348] T 5ac07d68002548958c4eeede92fcd4be P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:23.210783 22484 raft_consensus.cc:374] T 5ac07d68002548958c4eeede92fcd4be P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:23.210887 22484 raft_consensus.cc:729] T 5ac07d68002548958c4eeede92fcd4be P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:23.211134 22484 consensus_queue.cc:260] T 5ac07d68002548958c4eeede92fcd4be P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:23.211700 22484 ts_tablet_manager.cc:1398] T 5ac07d68002548958c4eeede92fcd4be P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:11:23.212756 22478 tablet_bootstrap.cc:492] T 7f0614c8c6ce4539925785c065829ad2 P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:23.212939 22478 ts_tablet_manager.cc:1367] T 7f0614c8c6ce4539925785c065829ad2 P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.004s
I20240503 08:11:23.213819 22478 raft_consensus.cc:348] T 7f0614c8c6ce4539925785c065829ad2 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.214119 22478 raft_consensus.cc:374] T 7f0614c8c6ce4539925785c065829ad2 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:23.214244 22478 raft_consensus.cc:729] T 7f0614c8c6ce4539925785c065829ad2 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:23.214416 22477 raft_consensus.cc:348] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.214627 22477 raft_consensus.cc:374] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:23.214780 22477 raft_consensus.cc:729] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:23.214883 22478 consensus_queue.cc:260] T 7f0614c8c6ce4539925785c065829ad2 P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.215292 22477 consensus_queue.cc:260] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.215459 22478 ts_tablet_manager.cc:1398] T 7f0614c8c6ce4539925785c065829ad2 P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:23.216859 22477 ts_tablet_manager.cc:1398] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:11:23.217186 22477 tablet_bootstrap.cc:492] T 7f0614c8c6ce4539925785c065829ad2 P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:23.218503 22477 tablet_bootstrap.cc:654] T 7f0614c8c6ce4539925785c065829ad2 P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:23.223312 22483 tablet_bootstrap.cc:492] T 5ac07d68002548958c4eeede92fcd4be P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:23.223515 22483 ts_tablet_manager.cc:1367] T 5ac07d68002548958c4eeede92fcd4be P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.008s
I20240503 08:11:23.224558 22483 raft_consensus.cc:348] T 5ac07d68002548958c4eeede92fcd4be P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:23.224834 22483 raft_consensus.cc:374] T 5ac07d68002548958c4eeede92fcd4be P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:23.224964 22483 raft_consensus.cc:729] T 5ac07d68002548958c4eeede92fcd4be P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:23.225340 22483 consensus_queue.cc:260] T 5ac07d68002548958c4eeede92fcd4be P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:23.225939 22483 ts_tablet_manager.cc:1398] T 5ac07d68002548958c4eeede92fcd4be P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 08:11:23.225960 22479 tablet_bootstrap.cc:492] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:23.226114 22479 ts_tablet_manager.cc:1367] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.006s
I20240503 08:11:23.227075 22479 raft_consensus.cc:348] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.227383 22479 raft_consensus.cc:374] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:23.227556 22479 raft_consensus.cc:729] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:23.227953 22479 consensus_queue.cc:260] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.229261 22479 ts_tablet_manager.cc:1398] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:23.230973 22477 tablet_bootstrap.cc:492] T 7f0614c8c6ce4539925785c065829ad2 P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:23.231254 22477 ts_tablet_manager.cc:1367] T 7f0614c8c6ce4539925785c065829ad2 P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20240503 08:11:23.232017 22477 raft_consensus.cc:348] T 7f0614c8c6ce4539925785c065829ad2 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.232237 22477 raft_consensus.cc:374] T 7f0614c8c6ce4539925785c065829ad2 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:23.232328 22477 raft_consensus.cc:729] T 7f0614c8c6ce4539925785c065829ad2 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:23.232621 22477 consensus_queue.cc:260] T 7f0614c8c6ce4539925785c065829ad2 P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.233233 22477 ts_tablet_manager.cc:1398] T 7f0614c8c6ce4539925785c065829ad2 P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:23.237224 22268 raft_consensus.cc:482] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:23.237404 22268 raft_consensus.cc:504] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.238042 22268 leader_election.cc:288] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6503743925434401a74aae303628e415 (127.20.107.134:42791), 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603)
I20240503 08:11:23.238437 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0614c8c6ce4539925785c065829ad2" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415" is_pre_election: true
I20240503 08:11:23.238653 21750 raft_consensus.cc:2454] T 7f0614c8c6ce4539925785c065829ad2 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 0.
I20240503 08:11:23.238968 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0614c8c6ce4539925785c065829ad2" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842" is_pre_election: true
I20240503 08:11:23.239012 21162 leader_election.cc:302] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:23.239136 21488 raft_consensus.cc:2454] T 7f0614c8c6ce4539925785c065829ad2 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 0.
I20240503 08:11:23.239238 22268 raft_consensus.cc:2781] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:23.239348 22268 raft_consensus.cc:482] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:23.239451 22268 raft_consensus.cc:3037] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:23.241333 22268 raft_consensus.cc:504] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.241796 22268 leader_election.cc:288] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 election: Requested vote from peers 6503743925434401a74aae303628e415 (127.20.107.134:42791), 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603)
I20240503 08:11:23.242208 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0614c8c6ce4539925785c065829ad2" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842"
I20240503 08:11:23.242278 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0614c8c6ce4539925785c065829ad2" candidate_uuid: "3b6eea641d214c79988ec6ad8821a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415"
I20240503 08:11:23.242406 21488 raft_consensus.cc:3037] T 7f0614c8c6ce4539925785c065829ad2 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:23.242499 21750 raft_consensus.cc:3037] T 7f0614c8c6ce4539925785c065829ad2 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:23.245167 21488 raft_consensus.cc:2454] T 7f0614c8c6ce4539925785c065829ad2 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 1.
I20240503 08:11:23.245185 21750 raft_consensus.cc:2454] T 7f0614c8c6ce4539925785c065829ad2 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6eea641d214c79988ec6ad8821a4bf in term 1.
I20240503 08:11:23.245582 21162 leader_election.cc:302] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:23.245882 22268 raft_consensus.cc:2781] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:23.246037 22268 raft_consensus.cc:686] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 LEADER]: Becoming Leader. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Running, Role: LEADER
I20240503 08:11:23.246336 22268 consensus_queue.cc:237] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.248944 20949 catalog_manager.cc:5484] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130). New cstate: current_term: 1 leader_uuid: "3b6eea641d214c79988ec6ad8821a4bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:23.254365 21981 raft_consensus.cc:482] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:23.254528 21981 raft_consensus.cc:504] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:23.255153 21981 leader_election.cc:288] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:23.255702 21618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac07d68002548958c4eeede92fcd4be" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" is_pre_election: true
I20240503 08:11:23.255791 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac07d68002548958c4eeede92fcd4be" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd" is_pre_election: true
I20240503 08:11:23.255921 21618 raft_consensus.cc:2454] T 5ac07d68002548958c4eeede92fcd4be P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 0.
I20240503 08:11:23.255980 21094 raft_consensus.cc:2454] T 5ac07d68002548958c4eeede92fcd4be P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 0.
I20240503 08:11:23.256376 21292 leader_election.cc:302] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:23.256685 21981 raft_consensus.cc:2781] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:23.256767 21981 raft_consensus.cc:482] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:23.256887 21981 raft_consensus.cc:3037] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:23.258639 21981 raft_consensus.cc:504] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:23.259181 21981 leader_election.cc:288] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 election: Requested vote from peers 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:23.259701 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac07d68002548958c4eeede92fcd4be" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd"
I20240503 08:11:23.259917 21094 raft_consensus.cc:3037] T 5ac07d68002548958c4eeede92fcd4be P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:23.261768 21094 raft_consensus.cc:2454] T 5ac07d68002548958c4eeede92fcd4be P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 1.
I20240503 08:11:23.262193 21291 leader_election.cc:302] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d9776f5d634ec2b167dc225659e4fd, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:23.262259 21618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac07d68002548958c4eeede92fcd4be" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
I20240503 08:11:23.262423 21981 raft_consensus.cc:2781] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:23.262480 21618 raft_consensus.cc:3037] T 5ac07d68002548958c4eeede92fcd4be P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:23.262849 21981 raft_consensus.cc:686] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 LEADER]: Becoming Leader. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Running, Role: LEADER
I20240503 08:11:23.263446 21981 consensus_queue.cc:237] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:23.265250 21618 raft_consensus.cc:2454] T 5ac07d68002548958c4eeede92fcd4be P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 1.
I20240503 08:11:23.267047 20949 catalog_manager.cc:5484] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131). New cstate: current_term: 1 leader_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:23.339533 22376 raft_consensus.cc:482] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:23.339742 22376 raft_consensus.cc:504] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.340468 22376 leader_election.cc:288] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6503743925434401a74aae303628e415 (127.20.107.134:42791), 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525)
I20240503 08:11:23.340963 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28305e23ecff40e9bce63ccf908cecb5" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf" is_pre_election: true
I20240503 08:11:23.340975 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28305e23ecff40e9bce63ccf908cecb5" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415" is_pre_election: true
I20240503 08:11:23.341199 21750 raft_consensus.cc:2454] T 28305e23ecff40e9bce63ccf908cecb5 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 0.
I20240503 08:11:23.341190 21225 raft_consensus.cc:2454] T 28305e23ecff40e9bce63ccf908cecb5 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 0.
I20240503 08:11:23.341611 21424 leader_election.cc:302] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e04594b1a5344d9bfefa4de05d70842, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:23.341888 22376 raft_consensus.cc:2781] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:23.342000 22376 raft_consensus.cc:482] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:23.342092 22376 raft_consensus.cc:3037] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:23.344743 22376 raft_consensus.cc:504] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.345397 22376 leader_election.cc:288] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Requested vote from peers 6503743925434401a74aae303628e415 (127.20.107.134:42791), 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525)
I20240503 08:11:23.346004 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28305e23ecff40e9bce63ccf908cecb5" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf"
I20240503 08:11:23.346120 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28305e23ecff40e9bce63ccf908cecb5" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415"
I20240503 08:11:23.346206 21225 raft_consensus.cc:3037] T 28305e23ecff40e9bce63ccf908cecb5 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:23.346292 21750 raft_consensus.cc:3037] T 28305e23ecff40e9bce63ccf908cecb5 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:23.348716 21225 raft_consensus.cc:2454] T 28305e23ecff40e9bce63ccf908cecb5 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:23.349002 21750 raft_consensus.cc:2454] T 28305e23ecff40e9bce63ccf908cecb5 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:23.349148 21423 leader_election.cc:302] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 3e04594b1a5344d9bfefa4de05d70842; no voters:
I20240503 08:11:23.349493 22376 raft_consensus.cc:2781] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:23.349880 22376 raft_consensus.cc:686] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 LEADER]: Becoming Leader. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Running, Role: LEADER
I20240503 08:11:23.350198 22376 consensus_queue.cc:237] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } }
I20240503 08:11:23.353194 20949 catalog_manager.cc:5484] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132). New cstate: current_term: 1 leader_uuid: "3e04594b1a5344d9bfefa4de05d70842" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:23.698812 22268 consensus_queue.cc:1035] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:23.705698 22268 consensus_queue.cc:1035] T 7f0614c8c6ce4539925785c065829ad2 P 3b6eea641d214c79988ec6ad8821a4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:23.727748 22497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:23.728070 22497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:23.736297 22497 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 08:11:23.844014 21981 consensus_queue.cc:1035] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:23.856006 21981 consensus_queue.cc:1035] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:23.877207 22376 consensus_queue.cc:1035] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:23.898757 22495 consensus_queue.cc:1035] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:23.905733 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.129:39921, user_credentials={real_user=slave}} blocked reactor thread for 44753.6us
INSERT report
rows total: 2000
time total: 317.292 ms
time per row: 0.158646 ms
I20240503 08:11:24.205683 20945 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41046:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:24.217031 21598 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5baa3ecf264275b6006bcf0fef7ca9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=924f5afd4b5a4d0d8db45759cc24e0e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:24.217530 21598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5baa3ecf264275b6006bcf0fef7ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:24.217578 21073 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5baa3ecf264275b6006bcf0fef7ca9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=924f5afd4b5a4d0d8db45759cc24e0e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:24.218019 21073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5baa3ecf264275b6006bcf0fef7ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:24.218447 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5baa3ecf264275b6006bcf0fef7ca9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=924f5afd4b5a4d0d8db45759cc24e0e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:24.218830 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5baa3ecf264275b6006bcf0fef7ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:24.221366 21206 tablet_service.cc:1467] Processing CreateTablet for tablet be04a137b2074903a273b3020563c5c3 (DEFAULT_TABLE table=rebalance_test_table_8 [id=924f5afd4b5a4d0d8db45759cc24e0e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:24.221827 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be04a137b2074903a273b3020563c5c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:24.224203 21599 tablet_service.cc:1467] Processing CreateTablet for tablet a838cc18e64b4fd0be47f3c9625a209d (DEFAULT_TABLE table=rebalance_test_table_8 [id=924f5afd4b5a4d0d8db45759cc24e0e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:24.224741 21599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a838cc18e64b4fd0be47f3c9625a209d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:24.227217 22569 tablet_bootstrap.cc:492] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:24.229527 21468 tablet_service.cc:1467] Processing CreateTablet for tablet be04a137b2074903a273b3020563c5c3 (DEFAULT_TABLE table=rebalance_test_table_8 [id=924f5afd4b5a4d0d8db45759cc24e0e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:24.229562 21337 tablet_service.cc:1467] Processing CreateTablet for tablet a838cc18e64b4fd0be47f3c9625a209d (DEFAULT_TABLE table=rebalance_test_table_8 [id=924f5afd4b5a4d0d8db45759cc24e0e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:24.229943 21468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be04a137b2074903a273b3020563c5c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:24.229959 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a838cc18e64b4fd0be47f3c9625a209d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:24.233284 21730 tablet_service.cc:1467] Processing CreateTablet for tablet be04a137b2074903a273b3020563c5c3 (DEFAULT_TABLE table=rebalance_test_table_8 [id=924f5afd4b5a4d0d8db45759cc24e0e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:24.233727 21730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be04a137b2074903a273b3020563c5c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:24.237453 22569 tablet_bootstrap.cc:654] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:24.238488 21074 tablet_service.cc:1467] Processing CreateTablet for tablet a838cc18e64b4fd0be47f3c9625a209d (DEFAULT_TABLE table=rebalance_test_table_8 [id=924f5afd4b5a4d0d8db45759cc24e0e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:24.239037 21074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a838cc18e64b4fd0be47f3c9625a209d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:24.240960 22569 tablet_bootstrap.cc:492] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:24.241730 22569 ts_tablet_manager.cc:1367] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240503 08:11:24.241724 22570 tablet_bootstrap.cc:492] T be04a137b2074903a273b3020563c5c3 P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:24.242604 22569 raft_consensus.cc:348] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:24.242902 22571 tablet_bootstrap.cc:492] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:24.243100 22569 raft_consensus.cc:374] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:24.243227 22569 raft_consensus.cc:729] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:24.243495 22569 consensus_queue.cc:260] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:24.244206 22569 ts_tablet_manager.cc:1398] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:24.245230 22571 tablet_bootstrap.cc:654] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:24.246284 22570 tablet_bootstrap.cc:654] T be04a137b2074903a273b3020563c5c3 P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:24.252681 22575 tablet_bootstrap.cc:492] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:24.254520 22575 tablet_bootstrap.cc:654] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:24.263170 22569 tablet_bootstrap.cc:492] T a838cc18e64b4fd0be47f3c9625a209d P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:24.263489 22571 tablet_bootstrap.cc:492] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:24.263702 22571 ts_tablet_manager.cc:1367] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.009s
I20240503 08:11:24.264958 22569 tablet_bootstrap.cc:654] T a838cc18e64b4fd0be47f3c9625a209d P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:24.264851 22571 raft_consensus.cc:348] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:24.265128 22571 raft_consensus.cc:374] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:24.265225 22571 raft_consensus.cc:729] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:24.266142 22576 tablet_bootstrap.cc:492] T be04a137b2074903a273b3020563c5c3 P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:24.266749 22571 consensus_queue.cc:260] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:24.267452 22571 ts_tablet_manager.cc:1398] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:11:24.271025 22576 tablet_bootstrap.cc:654] T be04a137b2074903a273b3020563c5c3 P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:24.272588 22570 tablet_bootstrap.cc:492] T be04a137b2074903a273b3020563c5c3 P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:24.273241 22570 ts_tablet_manager.cc:1367] T be04a137b2074903a273b3020563c5c3 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.011s
I20240503 08:11:24.274032 22568 tablet_bootstrap.cc:492] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:24.275612 22575 tablet_bootstrap.cc:492] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:24.275825 22575 ts_tablet_manager.cc:1367] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.000s
I20240503 08:11:24.277567 22569 tablet_bootstrap.cc:492] T a838cc18e64b4fd0be47f3c9625a209d P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:24.277805 22569 ts_tablet_manager.cc:1367] T a838cc18e64b4fd0be47f3c9625a209d P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.009s
I20240503 08:11:24.278856 22570 raft_consensus.cc:348] T be04a137b2074903a273b3020563c5c3 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:24.279083 22570 raft_consensus.cc:374] T be04a137b2074903a273b3020563c5c3 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:24.276670 22575 raft_consensus.cc:348] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:24.279189 22570 raft_consensus.cc:729] T be04a137b2074903a273b3020563c5c3 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:24.279251 22575 raft_consensus.cc:374] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:24.279343 22575 raft_consensus.cc:729] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:24.279436 22570 consensus_queue.cc:260] T be04a137b2074903a273b3020563c5c3 P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:24.278617 22569 raft_consensus.cc:348] T a838cc18e64b4fd0be47f3c9625a209d P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:24.279734 22569 raft_consensus.cc:374] T a838cc18e64b4fd0be47f3c9625a209d P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:24.279917 22569 raft_consensus.cc:729] T a838cc18e64b4fd0be47f3c9625a209d P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:24.280246 22569 consensus_queue.cc:260] T a838cc18e64b4fd0be47f3c9625a209d P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:24.280961 22570 ts_tablet_manager.cc:1398] T be04a137b2074903a273b3020563c5c3 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:11:24.280910 22575 consensus_queue.cc:260] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:24.281512 22575 ts_tablet_manager.cc:1398] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:11:24.281890 22575 tablet_bootstrap.cc:492] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:24.281919 22568 tablet_bootstrap.cc:654] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:24.282788 22569 ts_tablet_manager.cc:1398] T a838cc18e64b4fd0be47f3c9625a209d P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 08:11:24.283654 22575 tablet_bootstrap.cc:654] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:24.288122 22568 tablet_bootstrap.cc:492] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:24.288939 22576 tablet_bootstrap.cc:492] T be04a137b2074903a273b3020563c5c3 P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:24.288959 22568 ts_tablet_manager.cc:1367] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20240503 08:11:24.289140 22576 ts_tablet_manager.cc:1367] T be04a137b2074903a273b3020563c5c3 P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.004s
I20240503 08:11:24.289916 22576 raft_consensus.cc:348] T be04a137b2074903a273b3020563c5c3 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:24.290175 22576 raft_consensus.cc:374] T be04a137b2074903a273b3020563c5c3 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:24.290097 22568 raft_consensus.cc:348] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:24.290300 22576 raft_consensus.cc:729] T be04a137b2074903a273b3020563c5c3 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:24.290428 22568 raft_consensus.cc:374] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:24.290570 22576 consensus_queue.cc:260] T be04a137b2074903a273b3020563c5c3 P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:24.290576 22568 raft_consensus.cc:729] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:24.292171 22568 consensus_queue.cc:260] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:24.294142 22568 ts_tablet_manager.cc:1398] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:11:24.294504 22568 tablet_bootstrap.cc:492] T a838cc18e64b4fd0be47f3c9625a209d P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:24.295696 22576 ts_tablet_manager.cc:1398] T be04a137b2074903a273b3020563c5c3 P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:11:24.297068 22568 tablet_bootstrap.cc:654] T a838cc18e64b4fd0be47f3c9625a209d P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:24.301681 22575 tablet_bootstrap.cc:492] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:24.301904 22575 ts_tablet_manager.cc:1367] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.010s
I20240503 08:11:24.301908 22568 tablet_bootstrap.cc:492] T a838cc18e64b4fd0be47f3c9625a209d P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:24.302052 22568 ts_tablet_manager.cc:1367] T a838cc18e64b4fd0be47f3c9625a209d P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240503 08:11:24.302680 22575 raft_consensus.cc:348] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:24.302917 22575 raft_consensus.cc:374] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:24.302824 22568 raft_consensus.cc:348] T a838cc18e64b4fd0be47f3c9625a209d P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:24.302994 22575 raft_consensus.cc:729] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:24.303025 22568 raft_consensus.cc:374] T a838cc18e64b4fd0be47f3c9625a209d P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:24.303141 22568 raft_consensus.cc:729] T a838cc18e64b4fd0be47f3c9625a209d P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:24.303274 22575 consensus_queue.cc:260] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:24.303417 22568 consensus_queue.cc:260] T a838cc18e64b4fd0be47f3c9625a209d P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:24.303825 22575 ts_tablet_manager.cc:1398] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:24.304692 22568 ts_tablet_manager.cc:1398] T a838cc18e64b4fd0be47f3c9625a209d P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:11:24.354637 22558 raft_consensus.cc:482] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:24.354833 22558 raft_consensus.cc:504] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:24.355535 22558 leader_election.cc:288] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:24.355979 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be04a137b2074903a273b3020563c5c3" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf" is_pre_election: true
I20240503 08:11:24.356029 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be04a137b2074903a273b3020563c5c3" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415" is_pre_election: true
I20240503 08:11:24.356256 21225 raft_consensus.cc:2454] T be04a137b2074903a273b3020563c5c3 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 0.
I20240503 08:11:24.356282 21750 raft_consensus.cc:2454] T be04a137b2074903a273b3020563c5c3 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 0.
I20240503 08:11:24.356714 21424 leader_election.cc:302] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e04594b1a5344d9bfefa4de05d70842, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:24.357005 22558 raft_consensus.cc:2781] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:24.357113 22558 raft_consensus.cc:482] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:24.357199 22558 raft_consensus.cc:3037] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:24.390681 22424 raft_consensus.cc:482] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:24.390893 22424 raft_consensus.cc:504] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:24.391605 22424 leader_election.cc:288] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:24.392290 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5baa3ecf264275b6006bcf0fef7ca9" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd" is_pre_election: true
I20240503 08:11:24.392311 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5baa3ecf264275b6006bcf0fef7ca9" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" is_pre_election: true
I20240503 08:11:24.392570 21357 raft_consensus.cc:2454] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 0.
I20240503 08:11:24.392568 21094 raft_consensus.cc:2454] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 0.
I20240503 08:11:24.393035 21553 leader_election.cc:302] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 18d9776f5d634ec2b167dc225659e4fd; no voters:
I20240503 08:11:24.393332 22424 raft_consensus.cc:2781] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:24.393435 22424 raft_consensus.cc:482] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:24.393543 22424 raft_consensus.cc:3037] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:24.411478 22558 raft_consensus.cc:504] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:24.411499 22424 raft_consensus.cc:504] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:24.412176 22558 leader_election.cc:288] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:24.412710 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5baa3ecf264275b6006bcf0fef7ca9" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5"
I20240503 08:11:24.412770 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be04a137b2074903a273b3020563c5c3" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf"
I20240503 08:11:24.412835 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be04a137b2074903a273b3020563c5c3" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415"
I20240503 08:11:24.412987 21357 raft_consensus.cc:3037] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:24.413046 21225 raft_consensus.cc:3037] T be04a137b2074903a273b3020563c5c3 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:24.413076 21750 raft_consensus.cc:3037] T be04a137b2074903a273b3020563c5c3 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:24.415877 22424 leader_election.cc:288] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 election: Requested vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:24.416442 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5baa3ecf264275b6006bcf0fef7ca9" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd"
I20240503 08:11:24.416640 21094 raft_consensus.cc:3037] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:24.426868 22531 raft_consensus.cc:482] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:24.427031 22531 raft_consensus.cc:504] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:24.427667 22531 leader_election.cc:288] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:24.428167 21618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a838cc18e64b4fd0be47f3c9625a209d" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" is_pre_election: true
I20240503 08:11:24.428170 21093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a838cc18e64b4fd0be47f3c9625a209d" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd" is_pre_election: true
I20240503 08:11:24.428368 21093 raft_consensus.cc:2454] T a838cc18e64b4fd0be47f3c9625a209d P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 0.
I20240503 08:11:24.428401 21618 raft_consensus.cc:2454] T a838cc18e64b4fd0be47f3c9625a209d P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 0.
I20240503 08:11:24.428800 21291 leader_election.cc:302] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d9776f5d634ec2b167dc225659e4fd, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:24.429057 22531 raft_consensus.cc:2781] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:24.429226 22531 raft_consensus.cc:482] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:24.429340 22531 raft_consensus.cc:3037] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:24.510614 21225 raft_consensus.cc:2454] T be04a137b2074903a273b3020563c5c3 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:24.510608 21750 raft_consensus.cc:2454] T be04a137b2074903a273b3020563c5c3 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:24.511199 21424 leader_election.cc:302] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e04594b1a5344d9bfefa4de05d70842, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:24.511430 21357 raft_consensus.cc:2454] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 1.
I20240503 08:11:24.510608 21094 raft_consensus.cc:2454] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 1.
I20240503 08:11:24.512110 21552 leader_election.cc:302] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:24.512480 22424 raft_consensus.cc:2781] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:24.514021 22531 raft_consensus.cc:504] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:24.514431 22558 raft_consensus.cc:2781] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:24.514654 22558 raft_consensus.cc:686] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 LEADER]: Becoming Leader. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Running, Role: LEADER
I20240503 08:11:24.515007 22558 consensus_queue.cc:237] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:24.515264 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a838cc18e64b4fd0be47f3c9625a209d" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd"
I20240503 08:11:24.515466 21094 raft_consensus.cc:3037] T a838cc18e64b4fd0be47f3c9625a209d P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:24.515967 22424 raft_consensus.cc:686] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 LEADER]: Becoming Leader. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Running, Role: LEADER
I20240503 08:11:24.516414 22531 leader_election.cc:288] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 election: Requested vote from peers 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:24.516806 22424 consensus_queue.cc:237] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:24.517027 21618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a838cc18e64b4fd0be47f3c9625a209d" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
I20240503 08:11:24.517274 21618 raft_consensus.cc:3037] T a838cc18e64b4fd0be47f3c9625a209d P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:24.521258 20948 catalog_manager.cc:5484] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133). New cstate: current_term: 1 leader_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:24.521670 20945 catalog_manager.cc:5484] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132). New cstate: current_term: 1 leader_uuid: "3e04594b1a5344d9bfefa4de05d70842" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:24.612376 21094 raft_consensus.cc:2454] T a838cc18e64b4fd0be47f3c9625a209d P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 1.
I20240503 08:11:24.612358 21618 raft_consensus.cc:2454] T a838cc18e64b4fd0be47f3c9625a209d P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 1.
I20240503 08:11:24.612882 21291 leader_election.cc:302] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d9776f5d634ec2b167dc225659e4fd, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:24.613240 22531 raft_consensus.cc:2781] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:24.613400 22531 raft_consensus.cc:686] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 LEADER]: Becoming Leader. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Running, Role: LEADER
I20240503 08:11:24.613700 22531 consensus_queue.cc:237] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:24.617555 20945 catalog_manager.cc:5484] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131). New cstate: current_term: 1 leader_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:24.987524 22558 consensus_queue.cc:1035] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:24.991698 22424 consensus_queue.cc:1035] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:24.996135 22376 consensus_queue.cc:1035] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:25.002704 22583 consensus_queue.cc:1035] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:25.041157 22531 consensus_queue.cc:1035] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:25.051824 22531 consensus_queue.cc:1035] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:25.071056 22587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:25.071416 22587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:25.079536 22587 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 275.99 ms
time per row: 0.137995 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:11:26.473335 20945 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41046:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:26.483403 21730 tablet_service.cc:1467] Processing CreateTablet for tablet d7c9e2406d734bb68b7732190dbdd535 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9d4e006c135947e0ac9e34cf29d129ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:26.483906 21730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c9e2406d734bb68b7732190dbdd535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.483880 21468 tablet_service.cc:1467] Processing CreateTablet for tablet d7c9e2406d734bb68b7732190dbdd535 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9d4e006c135947e0ac9e34cf29d129ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:26.484309 21468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c9e2406d734bb68b7732190dbdd535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.484565 21205 tablet_service.cc:1467] Processing CreateTablet for tablet 8c221fe52b8a4ce083bd4ecb3f906dc0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9d4e006c135947e0ac9e34cf29d129ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:26.484982 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c221fe52b8a4ce083bd4ecb3f906dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.485167 21337 tablet_service.cc:1467] Processing CreateTablet for tablet a9fdcbd5803e49668916fc8b4d06dbbf (DEFAULT_TABLE table=rebalance_test_table_9 [id=9d4e006c135947e0ac9e34cf29d129ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:26.485564 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fdcbd5803e49668916fc8b4d06dbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.489246 21467 tablet_service.cc:1467] Processing CreateTablet for tablet 8c221fe52b8a4ce083bd4ecb3f906dc0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9d4e006c135947e0ac9e34cf29d129ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:26.489346 21206 tablet_service.cc:1467] Processing CreateTablet for tablet d7c9e2406d734bb68b7732190dbdd535 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9d4e006c135947e0ac9e34cf29d129ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:26.489625 21467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c221fe52b8a4ce083bd4ecb3f906dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.489766 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c9e2406d734bb68b7732190dbdd535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.489887 21073 tablet_service.cc:1467] Processing CreateTablet for tablet a9fdcbd5803e49668916fc8b4d06dbbf (DEFAULT_TABLE table=rebalance_test_table_9 [id=9d4e006c135947e0ac9e34cf29d129ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:26.490310 21073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fdcbd5803e49668916fc8b4d06dbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.496109 21729 tablet_service.cc:1467] Processing CreateTablet for tablet 8c221fe52b8a4ce083bd4ecb3f906dc0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9d4e006c135947e0ac9e34cf29d129ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:26.496699 21729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c221fe52b8a4ce083bd4ecb3f906dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.499202 21599 tablet_service.cc:1467] Processing CreateTablet for tablet a9fdcbd5803e49668916fc8b4d06dbbf (DEFAULT_TABLE table=rebalance_test_table_9 [id=9d4e006c135947e0ac9e34cf29d129ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:26.499773 21599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fdcbd5803e49668916fc8b4d06dbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.501194 22665 tablet_bootstrap.cc:492] T a9fdcbd5803e49668916fc8b4d06dbbf P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:26.502816 22665 tablet_bootstrap.cc:654] T a9fdcbd5803e49668916fc8b4d06dbbf P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.505689 22663 tablet_bootstrap.cc:492] T a9fdcbd5803e49668916fc8b4d06dbbf P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:26.505915 22664 tablet_bootstrap.cc:492] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:26.507587 22663 tablet_bootstrap.cc:654] T a9fdcbd5803e49668916fc8b4d06dbbf P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.507932 22664 tablet_bootstrap.cc:654] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.508978 22662 tablet_bootstrap.cc:492] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:26.510352 22662 tablet_bootstrap.cc:654] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.512060 22663 tablet_bootstrap.cc:492] T a9fdcbd5803e49668916fc8b4d06dbbf P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:26.512207 22663 ts_tablet_manager.cc:1367] T a9fdcbd5803e49668916fc8b4d06dbbf P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:11:26.512810 22670 tablet_bootstrap.cc:492] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:26.512812 22663 raft_consensus.cc:348] T a9fdcbd5803e49668916fc8b4d06dbbf P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:26.513036 22663 raft_consensus.cc:374] T a9fdcbd5803e49668916fc8b4d06dbbf P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.513146 22663 raft_consensus.cc:729] T a9fdcbd5803e49668916fc8b4d06dbbf P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.513407 22663 consensus_queue.cc:260] T a9fdcbd5803e49668916fc8b4d06dbbf P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:26.514156 22663 ts_tablet_manager.cc:1398] T a9fdcbd5803e49668916fc8b4d06dbbf P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:26.515377 22662 tablet_bootstrap.cc:492] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:26.515555 22662 ts_tablet_manager.cc:1367] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:11:26.516307 22662 raft_consensus.cc:348] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:26.516546 22662 raft_consensus.cc:374] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.516688 22662 raft_consensus.cc:729] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.516988 22662 consensus_queue.cc:260] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:26.518306 22662 ts_tablet_manager.cc:1398] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:11:26.518620 22662 tablet_bootstrap.cc:492] T d7c9e2406d734bb68b7732190dbdd535 P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:26.520128 22662 tablet_bootstrap.cc:654] T d7c9e2406d734bb68b7732190dbdd535 P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.520449 22671 tablet_bootstrap.cc:492] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:26.520548 22670 tablet_bootstrap.cc:654] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.522821 22671 tablet_bootstrap.cc:654] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.525662 22664 tablet_bootstrap.cc:492] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:26.525843 22664 ts_tablet_manager.cc:1367] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20240503 08:11:26.526716 22664 raft_consensus.cc:348] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:26.527014 22664 raft_consensus.cc:374] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.527128 22664 raft_consensus.cc:729] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.527396 22665 tablet_bootstrap.cc:492] T a9fdcbd5803e49668916fc8b4d06dbbf P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:26.527444 22664 consensus_queue.cc:260] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:26.527587 22665 ts_tablet_manager.cc:1367] T a9fdcbd5803e49668916fc8b4d06dbbf P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.008s
I20240503 08:11:26.527635 22670 tablet_bootstrap.cc:492] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:26.527838 22670 ts_tablet_manager.cc:1367] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20240503 08:11:26.527899 22671 tablet_bootstrap.cc:492] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:26.528049 22671 ts_tablet_manager.cc:1367] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 08:11:26.528357 22665 raft_consensus.cc:348] T a9fdcbd5803e49668916fc8b4d06dbbf P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:26.528580 22665 raft_consensus.cc:374] T a9fdcbd5803e49668916fc8b4d06dbbf P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.528717 22665 raft_consensus.cc:729] T a9fdcbd5803e49668916fc8b4d06dbbf P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.529106 22664 ts_tablet_manager.cc:1398] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:26.529109 22665 consensus_queue.cc:260] T a9fdcbd5803e49668916fc8b4d06dbbf P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:26.529650 22670 raft_consensus.cc:348] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:26.529832 22665 ts_tablet_manager.cc:1398] T a9fdcbd5803e49668916fc8b4d06dbbf P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:26.529940 22670 raft_consensus.cc:374] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.530128 22670 raft_consensus.cc:729] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.529443 22664 tablet_bootstrap.cc:492] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:26.530457 22670 consensus_queue.cc:260] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:26.531065 22671 raft_consensus.cc:348] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:26.531370 22671 raft_consensus.cc:374] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.531509 22671 raft_consensus.cc:729] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.531966 22671 consensus_queue.cc:260] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:26.532246 22664 tablet_bootstrap.cc:654] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.532626 22670 ts_tablet_manager.cc:1398] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:11:26.532959 22670 tablet_bootstrap.cc:492] T d7c9e2406d734bb68b7732190dbdd535 P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:26.534785 22671 ts_tablet_manager.cc:1398] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:11:26.535079 22670 tablet_bootstrap.cc:654] T d7c9e2406d734bb68b7732190dbdd535 P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.539759 22664 tablet_bootstrap.cc:492] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:26.540268 22664 ts_tablet_manager.cc:1367] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 08:11:26.541162 22664 raft_consensus.cc:348] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:26.541395 22664 raft_consensus.cc:374] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.541474 22664 raft_consensus.cc:729] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.541679 22664 consensus_queue.cc:260] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:26.542130 22664 ts_tablet_manager.cc:1398] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:26.549336 22662 tablet_bootstrap.cc:492] T d7c9e2406d734bb68b7732190dbdd535 P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:26.549512 22662 ts_tablet_manager.cc:1367] T d7c9e2406d734bb68b7732190dbdd535 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240503 08:11:26.550168 22662 raft_consensus.cc:348] T d7c9e2406d734bb68b7732190dbdd535 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:26.550338 22662 raft_consensus.cc:374] T d7c9e2406d734bb68b7732190dbdd535 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.550407 22662 raft_consensus.cc:729] T d7c9e2406d734bb68b7732190dbdd535 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.550618 22662 consensus_queue.cc:260] T d7c9e2406d734bb68b7732190dbdd535 P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:26.551102 22662 ts_tablet_manager.cc:1398] T d7c9e2406d734bb68b7732190dbdd535 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:11:26.557282 22670 tablet_bootstrap.cc:492] T d7c9e2406d734bb68b7732190dbdd535 P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:26.557454 22670 ts_tablet_manager.cc:1367] T d7c9e2406d734bb68b7732190dbdd535 P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.003s
I20240503 08:11:26.558197 22670 raft_consensus.cc:348] T d7c9e2406d734bb68b7732190dbdd535 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:26.558434 22670 raft_consensus.cc:374] T d7c9e2406d734bb68b7732190dbdd535 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.558517 22670 raft_consensus.cc:729] T d7c9e2406d734bb68b7732190dbdd535 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.558744 22670 consensus_queue.cc:260] T d7c9e2406d734bb68b7732190dbdd535 P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:26.559219 22670 ts_tablet_manager.cc:1398] T d7c9e2406d734bb68b7732190dbdd535 P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:11:26.586717 22596 raft_consensus.cc:482] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:26.586928 22596 raft_consensus.cc:504] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:26.587760 22596 leader_election.cc:288] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:26.588248 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fdcbd5803e49668916fc8b4d06dbbf" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" is_pre_election: true
I20240503 08:11:26.588346 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fdcbd5803e49668916fc8b4d06dbbf" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd" is_pre_election: true
I20240503 08:11:26.588465 21357 raft_consensus.cc:2454] T a9fdcbd5803e49668916fc8b4d06dbbf P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 0.
I20240503 08:11:26.588626 21094 raft_consensus.cc:2454] T a9fdcbd5803e49668916fc8b4d06dbbf P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 0.
I20240503 08:11:26.588987 21552 leader_election.cc:302] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:26.589285 22596 raft_consensus.cc:2781] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:26.589401 22596 raft_consensus.cc:482] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:26.589489 22596 raft_consensus.cc:3037] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:26.591301 22596 raft_consensus.cc:504] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:26.591831 22596 leader_election.cc:288] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 election: Requested vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:26.592195 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fdcbd5803e49668916fc8b4d06dbbf" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5"
I20240503 08:11:26.592279 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fdcbd5803e49668916fc8b4d06dbbf" candidate_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd"
I20240503 08:11:26.592396 21357 raft_consensus.cc:3037] T a9fdcbd5803e49668916fc8b4d06dbbf P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:26.592496 21094 raft_consensus.cc:3037] T a9fdcbd5803e49668916fc8b4d06dbbf P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:26.594303 21357 raft_consensus.cc:2454] T a9fdcbd5803e49668916fc8b4d06dbbf P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 1.
I20240503 08:11:26.594394 21094 raft_consensus.cc:2454] T a9fdcbd5803e49668916fc8b4d06dbbf P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17af6d2a2fd14b6d93e232da6aa8f6a9 in term 1.
I20240503 08:11:26.594668 21552 leader_election.cc:302] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:26.594935 22596 raft_consensus.cc:2781] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:26.595345 22596 raft_consensus.cc:686] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 LEADER]: Becoming Leader. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Running, Role: LEADER
I20240503 08:11:26.595666 22596 consensus_queue.cc:237] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:26.598735 20945 catalog_manager.cc:5484] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133). New cstate: current_term: 1 leader_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:26.640909 21839 raft_consensus.cc:482] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:26.641109 21839 raft_consensus.cc:504] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:26.641700 21839 leader_election.cc:288] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603)
I20240503 08:11:26.642154 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c221fe52b8a4ce083bd4ecb3f906dc0" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf" is_pre_election: true
I20240503 08:11:26.642253 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c221fe52b8a4ce083bd4ecb3f906dc0" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842" is_pre_election: true
I20240503 08:11:26.642410 21225 raft_consensus.cc:2454] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 0.
I20240503 08:11:26.642490 21488 raft_consensus.cc:2454] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 0.
I20240503 08:11:26.642858 21685 leader_election.cc:302] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:26.643194 21839 raft_consensus.cc:2781] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:26.643285 21839 raft_consensus.cc:482] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:26.643370 21839 raft_consensus.cc:3037] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:26.645237 21839 raft_consensus.cc:504] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:26.645800 21839 leader_election.cc:288] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603)
I20240503 08:11:26.646282 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c221fe52b8a4ce083bd4ecb3f906dc0" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842"
I20240503 08:11:26.646286 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c221fe52b8a4ce083bd4ecb3f906dc0" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf"
I20240503 08:11:26.646497 21225 raft_consensus.cc:3037] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:26.646510 21488 raft_consensus.cc:3037] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:26.649298 21488 raft_consensus.cc:2454] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 1.
I20240503 08:11:26.649314 21225 raft_consensus.cc:2454] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 1.
I20240503 08:11:26.649739 21684 leader_election.cc:302] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e04594b1a5344d9bfefa4de05d70842, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:26.649973 21839 raft_consensus.cc:2781] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:26.650354 21839 raft_consensus.cc:686] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [term 1 LEADER]: Becoming Leader. State: Replica: 6503743925434401a74aae303628e415, State: Running, Role: LEADER
I20240503 08:11:26.650622 21839 consensus_queue.cc:237] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } }
I20240503 08:11:26.653695 20945 catalog_manager.cc:5484] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6503743925434401a74aae303628e415 (127.20.107.134). New cstate: current_term: 1 leader_uuid: "6503743925434401a74aae303628e415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:26.681208 22558 raft_consensus.cc:482] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:26.681397 22558 raft_consensus.cc:504] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:26.682044 22558 leader_election.cc:288] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6503743925434401a74aae303628e415 (127.20.107.134:42791), 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525)
I20240503 08:11:26.682606 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c9e2406d734bb68b7732190dbdd535" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf" is_pre_election: true
I20240503 08:11:26.682861 21225 raft_consensus.cc:2454] T d7c9e2406d734bb68b7732190dbdd535 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 0.
I20240503 08:11:26.682837 21749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c9e2406d734bb68b7732190dbdd535" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415" is_pre_election: true
I20240503 08:11:26.683036 21749 raft_consensus.cc:2454] T d7c9e2406d734bb68b7732190dbdd535 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 0.
I20240503 08:11:26.683270 21423 leader_election.cc:302] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 3e04594b1a5344d9bfefa4de05d70842; no voters:
I20240503 08:11:26.683552 22681 raft_consensus.cc:2781] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:26.683650 22681 raft_consensus.cc:482] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:26.683758 22681 raft_consensus.cc:3037] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:26.685837 22681 raft_consensus.cc:504] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:26.686376 22681 leader_election.cc:288] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Requested vote from peers 6503743925434401a74aae303628e415 (127.20.107.134:42791), 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525)
I20240503 08:11:26.686769 21749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c9e2406d734bb68b7732190dbdd535" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415"
I20240503 08:11:26.686811 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c9e2406d734bb68b7732190dbdd535" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf"
I20240503 08:11:26.686986 21749 raft_consensus.cc:3037] T d7c9e2406d734bb68b7732190dbdd535 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:26.687034 21225 raft_consensus.cc:3037] T d7c9e2406d734bb68b7732190dbdd535 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:26.689816 21749 raft_consensus.cc:2454] T d7c9e2406d734bb68b7732190dbdd535 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:26.689824 21225 raft_consensus.cc:2454] T d7c9e2406d734bb68b7732190dbdd535 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:26.690218 21423 leader_election.cc:302] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 3e04594b1a5344d9bfefa4de05d70842; no voters:
I20240503 08:11:26.690511 22681 raft_consensus.cc:2781] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:26.690645 22681 raft_consensus.cc:686] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 LEADER]: Becoming Leader. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Running, Role: LEADER
I20240503 08:11:26.690903 22681 consensus_queue.cc:237] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:26.693858 20945 catalog_manager.cc:5484] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132). New cstate: current_term: 1 leader_uuid: "3e04594b1a5344d9bfefa4de05d70842" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:27.038821 21839 consensus_queue.cc:1035] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:27.048015 22685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:27.048342 22685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:11:27.057575 22679 consensus_queue.cc:1035] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:27.057834 22685 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 08:11:27.113180 22681 consensus_queue.cc:1035] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:27.118125 22681 consensus_queue.cc:1035] T d7c9e2406d734bb68b7732190dbdd535 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:27.173766 22596 consensus_queue.cc:1035] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:27.206025 22677 consensus_queue.cc:1035] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:27.317430 21553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.129:39921, user_credentials={real_user=slave}} blocked reactor thread for 44281.5us
INSERT report
rows total: 2000
time total: 295.5 ms
time per row: 0.14775 ms
I20240503 08:11:27.513121 20946 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41046:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:27.522109 21599 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5c3da0019f48dd8592eb5b5d393256 (DEFAULT_TABLE table=rebalance_test_table_10 [id=03c7caad68dc47a181f85e1dc49d17b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:27.522398 21073 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5c3da0019f48dd8592eb5b5d393256 (DEFAULT_TABLE table=rebalance_test_table_10 [id=03c7caad68dc47a181f85e1dc49d17b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:27.522611 21599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5c3da0019f48dd8592eb5b5d393256. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:27.522948 21073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5c3da0019f48dd8592eb5b5d393256. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:27.523248 21467 tablet_service.cc:1467] Processing CreateTablet for tablet 95f72713a7634db0a84ba8c144148db5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=03c7caad68dc47a181f85e1dc49d17b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:27.523713 21467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f72713a7634db0a84ba8c144148db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:27.525297 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 4e77b4ff1be549068fe4ebb4de6d46a1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=03c7caad68dc47a181f85e1dc49d17b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:27.525295 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 95f72713a7634db0a84ba8c144148db5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=03c7caad68dc47a181f85e1dc49d17b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:27.525700 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f72713a7634db0a84ba8c144148db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:27.525717 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e77b4ff1be549068fe4ebb4de6d46a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:27.529342 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5c3da0019f48dd8592eb5b5d393256 (DEFAULT_TABLE table=rebalance_test_table_10 [id=03c7caad68dc47a181f85e1dc49d17b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:27.529369 21729 tablet_service.cc:1467] Processing CreateTablet for tablet 95f72713a7634db0a84ba8c144148db5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=03c7caad68dc47a181f85e1dc49d17b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:27.529767 21729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f72713a7634db0a84ba8c144148db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:27.529815 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5c3da0019f48dd8592eb5b5d393256. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:27.533741 21074 tablet_service.cc:1467] Processing CreateTablet for tablet 4e77b4ff1be549068fe4ebb4de6d46a1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=03c7caad68dc47a181f85e1dc49d17b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:27.534262 21074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e77b4ff1be549068fe4ebb4de6d46a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:27.538045 21598 tablet_service.cc:1467] Processing CreateTablet for tablet 4e77b4ff1be549068fe4ebb4de6d46a1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=03c7caad68dc47a181f85e1dc49d17b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:27.538614 21598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e77b4ff1be549068fe4ebb4de6d46a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:27.539474 22756 tablet_bootstrap.cc:492] T 95f72713a7634db0a84ba8c144148db5 P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:27.541802 22756 tablet_bootstrap.cc:654] T 95f72713a7634db0a84ba8c144148db5 P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:27.551826 22759 tablet_bootstrap.cc:492] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:27.554836 22759 tablet_bootstrap.cc:654] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:27.554848 22758 tablet_bootstrap.cc:492] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:27.558161 22758 tablet_bootstrap.cc:654] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:27.560586 22756 tablet_bootstrap.cc:492] T 95f72713a7634db0a84ba8c144148db5 P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:27.560809 22756 ts_tablet_manager.cc:1367] T 95f72713a7634db0a84ba8c144148db5 P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.008s
I20240503 08:11:27.561691 22756 raft_consensus.cc:348] T 95f72713a7634db0a84ba8c144148db5 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:27.562043 22756 raft_consensus.cc:374] T 95f72713a7634db0a84ba8c144148db5 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:27.562202 22756 raft_consensus.cc:729] T 95f72713a7634db0a84ba8c144148db5 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:27.562521 22756 consensus_queue.cc:260] T 95f72713a7634db0a84ba8c144148db5 P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:27.564682 22762 tablet_bootstrap.cc:492] T 95f72713a7634db0a84ba8c144148db5 P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:27.565057 22756 ts_tablet_manager.cc:1398] T 95f72713a7634db0a84ba8c144148db5 P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:11:27.567865 22762 tablet_bootstrap.cc:654] T 95f72713a7634db0a84ba8c144148db5 P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:27.568573 22763 tablet_bootstrap.cc:492] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:27.571141 22763 tablet_bootstrap.cc:654] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:27.573823 22758 tablet_bootstrap.cc:492] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:27.574036 22758 ts_tablet_manager.cc:1367] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.011s
I20240503 08:11:27.575172 22758 raft_consensus.cc:348] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:27.575490 22759 tablet_bootstrap.cc:492] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:27.575551 22758 raft_consensus.cc:374] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:27.575727 22758 raft_consensus.cc:729] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:27.576048 22758 consensus_queue.cc:260] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:27.576509 22755 tablet_bootstrap.cc:492] T 4b5c3da0019f48dd8592eb5b5d393256 P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:27.576557 22759 ts_tablet_manager.cc:1367] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.004s
I20240503 08:11:27.576911 22762 tablet_bootstrap.cc:492] T 95f72713a7634db0a84ba8c144148db5 P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:27.577121 22762 ts_tablet_manager.cc:1367] T 95f72713a7634db0a84ba8c144148db5 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20240503 08:11:27.577486 22763 tablet_bootstrap.cc:492] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:27.577486 22759 raft_consensus.cc:348] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:27.577735 22759 raft_consensus.cc:374] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:27.577865 22759 raft_consensus.cc:729] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:27.577948 22762 raft_consensus.cc:348] T 95f72713a7634db0a84ba8c144148db5 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:27.578090 22763 ts_tablet_manager.cc:1367] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 08:11:27.578153 22759 consensus_queue.cc:260] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:27.578303 22762 raft_consensus.cc:374] T 95f72713a7634db0a84ba8c144148db5 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:27.578433 22762 raft_consensus.cc:729] T 95f72713a7634db0a84ba8c144148db5 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:27.578754 22762 consensus_queue.cc:260] T 95f72713a7634db0a84ba8c144148db5 P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:27.578876 22763 raft_consensus.cc:348] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:27.579658 22759 ts_tablet_manager.cc:1398] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:27.579979 22759 tablet_bootstrap.cc:492] T 4b5c3da0019f48dd8592eb5b5d393256 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:27.580205 22762 ts_tablet_manager.cc:1398] T 95f72713a7634db0a84ba8c144148db5 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:11:27.580363 22763 raft_consensus.cc:374] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:27.580485 22763 raft_consensus.cc:729] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:27.581346 22755 tablet_bootstrap.cc:654] T 4b5c3da0019f48dd8592eb5b5d393256 P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:27.580746 22763 consensus_queue.cc:260] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:27.581601 22759 tablet_bootstrap.cc:654] T 4b5c3da0019f48dd8592eb5b5d393256 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:27.583781 22758 ts_tablet_manager.cc:1398] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 08:11:27.584138 22758 tablet_bootstrap.cc:492] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:27.584952 22763 ts_tablet_manager.cc:1398] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 08:11:27.589059 22758 tablet_bootstrap.cc:654] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:27.589988 22693 raft_consensus.cc:482] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:27.590130 22693 raft_consensus.cc:504] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:27.591056 22693 leader_election.cc:288] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603)
I20240503 08:11:27.591905 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f72713a7634db0a84ba8c144148db5" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf" is_pre_election: true
I20240503 08:11:27.592150 21225 raft_consensus.cc:2454] T 95f72713a7634db0a84ba8c144148db5 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 0.
I20240503 08:11:27.592379 22755 tablet_bootstrap.cc:492] T 4b5c3da0019f48dd8592eb5b5d393256 P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:27.592530 22755 ts_tablet_manager.cc:1367] T 4b5c3da0019f48dd8592eb5b5d393256 P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240503 08:11:27.593274 21685 leader_election.cc:302] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:27.592787 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f72713a7634db0a84ba8c144148db5" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842" is_pre_election: true
I20240503 08:11:27.593434 22755 raft_consensus.cc:348] T 4b5c3da0019f48dd8592eb5b5d393256 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:27.593636 22755 raft_consensus.cc:374] T 4b5c3da0019f48dd8592eb5b5d393256 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:27.593695 21488 raft_consensus.cc:2454] T 95f72713a7634db0a84ba8c144148db5 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 0.
I20240503 08:11:27.593727 22755 raft_consensus.cc:729] T 4b5c3da0019f48dd8592eb5b5d393256 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:27.593964 22755 consensus_queue.cc:260] T 4b5c3da0019f48dd8592eb5b5d393256 P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:27.594380 22693 raft_consensus.cc:2781] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:27.594491 22693 raft_consensus.cc:482] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:27.594585 22693 raft_consensus.cc:3037] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:27.594609 22755 ts_tablet_manager.cc:1398] T 4b5c3da0019f48dd8592eb5b5d393256 P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:27.594640 22759 tablet_bootstrap.cc:492] T 4b5c3da0019f48dd8592eb5b5d393256 P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:27.594835 22759 ts_tablet_manager.cc:1367] T 4b5c3da0019f48dd8592eb5b5d393256 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20240503 08:11:27.595057 22755 tablet_bootstrap.cc:492] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:27.595742 22759 raft_consensus.cc:348] T 4b5c3da0019f48dd8592eb5b5d393256 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:27.596021 22759 raft_consensus.cc:374] T 4b5c3da0019f48dd8592eb5b5d393256 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:27.596105 22759 raft_consensus.cc:729] T 4b5c3da0019f48dd8592eb5b5d393256 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:27.596387 22759 consensus_queue.cc:260] T 4b5c3da0019f48dd8592eb5b5d393256 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:27.597007 22759 ts_tablet_manager.cc:1398] T 4b5c3da0019f48dd8592eb5b5d393256 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:27.597254 22693 raft_consensus.cc:504] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:27.598160 22755 tablet_bootstrap.cc:654] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:27.598217 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f72713a7634db0a84ba8c144148db5" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf"
I20240503 08:11:27.598428 21225 raft_consensus.cc:3037] T 95f72713a7634db0a84ba8c144148db5 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:27.600180 22758 tablet_bootstrap.cc:492] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:27.600324 22758 ts_tablet_manager.cc:1367] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.007s
I20240503 08:11:27.601154 22758 raft_consensus.cc:348] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:27.601430 22758 raft_consensus.cc:374] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:27.601555 22758 raft_consensus.cc:729] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:27.601866 22758 consensus_queue.cc:260] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:27.603058 22758 ts_tablet_manager.cc:1398] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:27.603266 21225 raft_consensus.cc:2454] T 95f72713a7634db0a84ba8c144148db5 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 1.
I20240503 08:11:27.603879 21685 leader_election.cc:302] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:27.604287 22679 raft_consensus.cc:2781] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:27.604445 22679 raft_consensus.cc:686] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [term 1 LEADER]: Becoming Leader. State: Replica: 6503743925434401a74aae303628e415, State: Running, Role: LEADER
I20240503 08:11:27.604795 22679 consensus_queue.cc:237] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:27.605510 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f72713a7634db0a84ba8c144148db5" candidate_uuid: "6503743925434401a74aae303628e415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e04594b1a5344d9bfefa4de05d70842"
I20240503 08:11:27.605787 21488 raft_consensus.cc:3037] T 95f72713a7634db0a84ba8c144148db5 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:27.604454 22693 leader_election.cc:288] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603)
I20240503 08:11:27.609160 21488 raft_consensus.cc:2454] T 95f72713a7634db0a84ba8c144148db5 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6503743925434401a74aae303628e415 in term 1.
I20240503 08:11:27.610056 22755 tablet_bootstrap.cc:492] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:27.610224 22755 ts_tablet_manager.cc:1367] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.010s
I20240503 08:11:27.610033 20946 catalog_manager.cc:5484] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6503743925434401a74aae303628e415 (127.20.107.134). New cstate: current_term: 1 leader_uuid: "6503743925434401a74aae303628e415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:27.610865 22755 raft_consensus.cc:348] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:27.611096 22755 raft_consensus.cc:374] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:27.611186 22755 raft_consensus.cc:729] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:27.611461 22755 consensus_queue.cc:260] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:27.612030 22755 ts_tablet_manager.cc:1398] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:27.641090 22641 raft_consensus.cc:482] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:27.641292 22641 raft_consensus.cc:504] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:27.641957 22641 leader_election.cc:288] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:27.642513 21094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e77b4ff1be549068fe4ebb4de6d46a1" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd" is_pre_election: true
I20240503 08:11:27.642534 21618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e77b4ff1be549068fe4ebb4de6d46a1" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" is_pre_election: true
I20240503 08:11:27.642772 21618 raft_consensus.cc:2454] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 0.
I20240503 08:11:27.642779 21094 raft_consensus.cc:2454] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 0.
I20240503 08:11:27.643189 21292 leader_election.cc:302] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:27.643474 22641 raft_consensus.cc:2781] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:27.643611 22641 raft_consensus.cc:482] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:27.643744 22641 raft_consensus.cc:3037] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:27.645889 22641 raft_consensus.cc:504] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:27.646450 22641 leader_election.cc:288] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 election: Requested vote from peers 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:27.646896 21618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e77b4ff1be549068fe4ebb4de6d46a1" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
I20240503 08:11:27.646910 21093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e77b4ff1be549068fe4ebb4de6d46a1" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd"
I20240503 08:11:27.647105 21093 raft_consensus.cc:3037] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:27.647109 21618 raft_consensus.cc:3037] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:27.650023 21618 raft_consensus.cc:2454] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 1.
I20240503 08:11:27.650044 21093 raft_consensus.cc:2454] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 1.
I20240503 08:11:27.650455 21292 leader_election.cc:302] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:27.650768 22641 raft_consensus.cc:2781] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:27.650908 22641 raft_consensus.cc:686] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 LEADER]: Becoming Leader. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Running, Role: LEADER
I20240503 08:11:27.651268 22641 consensus_queue.cc:237] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } }
I20240503 08:11:27.654296 20949 catalog_manager.cc:5484] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131). New cstate: current_term: 1 leader_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:27.846660 22641 raft_consensus.cc:482] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:27.846861 22641 raft_consensus.cc:504] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:27.847488 22641 leader_election.cc:288] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:27.847903 21618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5c3da0019f48dd8592eb5b5d393256" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" is_pre_election: true
I20240503 08:11:27.848112 21618 raft_consensus.cc:2454] T 4b5c3da0019f48dd8592eb5b5d393256 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 0.
I20240503 08:11:27.848023 21093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5c3da0019f48dd8592eb5b5d393256" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd" is_pre_election: true
I20240503 08:11:27.848323 21093 raft_consensus.cc:2454] T 4b5c3da0019f48dd8592eb5b5d393256 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 0.
I20240503 08:11:27.848475 21292 leader_election.cc:302] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:27.848762 22641 raft_consensus.cc:2781] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:27.848847 22641 raft_consensus.cc:482] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:27.848974 22641 raft_consensus.cc:3037] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:27.850909 22641 raft_consensus.cc:504] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:27.851461 22641 leader_election.cc:288] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 election: Requested vote from peers 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423), 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921)
I20240503 08:11:27.851783 21618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5c3da0019f48dd8592eb5b5d393256" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
I20240503 08:11:27.851878 21093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5c3da0019f48dd8592eb5b5d393256" candidate_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d9776f5d634ec2b167dc225659e4fd"
I20240503 08:11:27.851990 21618 raft_consensus.cc:3037] T 4b5c3da0019f48dd8592eb5b5d393256 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:27.852087 21093 raft_consensus.cc:3037] T 4b5c3da0019f48dd8592eb5b5d393256 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:27.854146 21618 raft_consensus.cc:2454] T 4b5c3da0019f48dd8592eb5b5d393256 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 1.
I20240503 08:11:27.854247 21093 raft_consensus.cc:2454] T 4b5c3da0019f48dd8592eb5b5d393256 P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8d441f0f6425b88ec78c8b0d7f3d5 in term 1.
I20240503 08:11:27.854513 21292 leader_election.cc:302] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17af6d2a2fd14b6d93e232da6aa8f6a9, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:27.854782 22641 raft_consensus.cc:2781] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:27.854951 22641 raft_consensus.cc:686] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 LEADER]: Becoming Leader. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Running, Role: LEADER
I20240503 08:11:27.855206 22641 consensus_queue.cc:237] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } }
I20240503 08:11:27.864111 20949 catalog_manager.cc:5484] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131). New cstate: current_term: 1 leader_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:27.986510 22679 consensus_queue.cc:1035] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:27.995075 21839 consensus_queue.cc:1035] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:28.092226 22641 consensus_queue.cc:1035] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:28.101029 22783 consensus_queue.cc:1035] T 4e77b4ff1be549068fe4ebb4de6d46a1 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:28.254524 22783 consensus_queue.cc:1035] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:28.258771 22774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:28.259212 22774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:11:28.260753 22641 consensus_queue.cc:1035] T 4b5c3da0019f48dd8592eb5b5d393256 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:28.272423 22774 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
INSERT report
rows total: 2000
time total: 270.783 ms
time per row: 0.135392 ms
I20240503 08:11:28.703459 20949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41046:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:28.712417 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 37d7d0f4ab374b6fac1b537e54b030a6 (DEFAULT_TABLE table=rebalance_test_table_11 [id=961d19d3b2654fe6bd9c2a0b2a570a6c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:28.712950 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d7d0f4ab374b6fac1b537e54b030a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:28.712985 21729 tablet_service.cc:1467] Processing CreateTablet for tablet 37d7d0f4ab374b6fac1b537e54b030a6 (DEFAULT_TABLE table=rebalance_test_table_11 [id=961d19d3b2654fe6bd9c2a0b2a570a6c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:28.713378 21729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d7d0f4ab374b6fac1b537e54b030a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:28.713299 21468 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa3476fa91846358562fa4425410d16 (DEFAULT_TABLE table=rebalance_test_table_11 [id=961d19d3b2654fe6bd9c2a0b2a570a6c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:28.713527 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 483431ac3a40425b90b051d72c5c3453 (DEFAULT_TABLE table=rebalance_test_table_11 [id=961d19d3b2654fe6bd9c2a0b2a570a6c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:28.713706 21468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aa3476fa91846358562fa4425410d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:28.713943 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 483431ac3a40425b90b051d72c5c3453. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:28.717255 21467 tablet_service.cc:1467] Processing CreateTablet for tablet 37d7d0f4ab374b6fac1b537e54b030a6 (DEFAULT_TABLE table=rebalance_test_table_11 [id=961d19d3b2654fe6bd9c2a0b2a570a6c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:28.717453 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa3476fa91846358562fa4425410d16 (DEFAULT_TABLE table=rebalance_test_table_11 [id=961d19d3b2654fe6bd9c2a0b2a570a6c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:28.717619 21467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d7d0f4ab374b6fac1b537e54b030a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:28.717823 21730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aa3476fa91846358562fa4425410d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:28.721777 21073 tablet_service.cc:1467] Processing CreateTablet for tablet 483431ac3a40425b90b051d72c5c3453 (DEFAULT_TABLE table=rebalance_test_table_11 [id=961d19d3b2654fe6bd9c2a0b2a570a6c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:28.722352 21073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 483431ac3a40425b90b051d72c5c3453. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:28.723409 21599 tablet_service.cc:1467] Processing CreateTablet for tablet 483431ac3a40425b90b051d72c5c3453 (DEFAULT_TABLE table=rebalance_test_table_11 [id=961d19d3b2654fe6bd9c2a0b2a570a6c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:28.723659 21205 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa3476fa91846358562fa4425410d16 (DEFAULT_TABLE table=rebalance_test_table_11 [id=961d19d3b2654fe6bd9c2a0b2a570a6c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:28.723832 21599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 483431ac3a40425b90b051d72c5c3453. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:28.724071 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aa3476fa91846358562fa4425410d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:28.740046 22845 tablet_bootstrap.cc:492] T 483431ac3a40425b90b051d72c5c3453 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Bootstrap starting.
I20240503 08:11:28.741475 22847 tablet_bootstrap.cc:492] T 37d7d0f4ab374b6fac1b537e54b030a6 P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:28.742213 22844 tablet_bootstrap.cc:492] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd: Bootstrap starting.
I20240503 08:11:28.744544 22845 tablet_bootstrap.cc:654] T 483431ac3a40425b90b051d72c5c3453 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:28.746011 22841 tablet_bootstrap.cc:492] T 483431ac3a40425b90b051d72c5c3453 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Bootstrap starting.
I20240503 08:11:28.746977 22844 tablet_bootstrap.cc:654] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:28.747560 22847 tablet_bootstrap.cc:654] T 37d7d0f4ab374b6fac1b537e54b030a6 P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:28.747715 22843 tablet_bootstrap.cc:492] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:28.750409 22846 tablet_bootstrap.cc:492] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:28.751796 22841 tablet_bootstrap.cc:654] T 483431ac3a40425b90b051d72c5c3453 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:28.751870 22843 tablet_bootstrap.cc:654] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:28.755599 22845 tablet_bootstrap.cc:492] T 483431ac3a40425b90b051d72c5c3453 P 17af6d2a2fd14b6d93e232da6aa8f6a9: No bootstrap required, opened a new log
I20240503 08:11:28.755898 22845 ts_tablet_manager.cc:1367] T 483431ac3a40425b90b051d72c5c3453 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240503 08:11:28.757094 22845 raft_consensus.cc:348] T 483431ac3a40425b90b051d72c5c3453 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:28.757499 22845 raft_consensus.cc:374] T 483431ac3a40425b90b051d72c5c3453 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:28.757664 22845 raft_consensus.cc:729] T 483431ac3a40425b90b051d72c5c3453 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af6d2a2fd14b6d93e232da6aa8f6a9, State: Initialized, Role: FOLLOWER
I20240503 08:11:28.757761 22846 tablet_bootstrap.cc:654] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:28.758090 22845 consensus_queue.cc:260] T 483431ac3a40425b90b051d72c5c3453 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:28.759737 22845 ts_tablet_manager.cc:1398] T 483431ac3a40425b90b051d72c5c3453 P 17af6d2a2fd14b6d93e232da6aa8f6a9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:11:28.759747 22841 tablet_bootstrap.cc:492] T 483431ac3a40425b90b051d72c5c3453 P 74c8d441f0f6425b88ec78c8b0d7f3d5: No bootstrap required, opened a new log
I20240503 08:11:28.759909 22841 ts_tablet_manager.cc:1367] T 483431ac3a40425b90b051d72c5c3453 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20240503 08:11:28.761116 22847 tablet_bootstrap.cc:492] T 37d7d0f4ab374b6fac1b537e54b030a6 P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:28.761495 22847 ts_tablet_manager.cc:1367] T 37d7d0f4ab374b6fac1b537e54b030a6 P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.006s
I20240503 08:11:28.761557 22844 tablet_bootstrap.cc:492] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd: No bootstrap required, opened a new log
I20240503 08:11:28.761799 22844 ts_tablet_manager.cc:1367] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20240503 08:11:28.762987 22844 raft_consensus.cc:348] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:28.763324 22844 raft_consensus.cc:374] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:28.763458 22844 raft_consensus.cc:729] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Initialized, Role: FOLLOWER
I20240503 08:11:28.764093 22844 consensus_queue.cc:260] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:28.764247 22847 raft_consensus.cc:348] T 37d7d0f4ab374b6fac1b537e54b030a6 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.764652 22847 raft_consensus.cc:374] T 37d7d0f4ab374b6fac1b537e54b030a6 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:28.764847 22847 raft_consensus.cc:729] T 37d7d0f4ab374b6fac1b537e54b030a6 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:28.765405 22844 ts_tablet_manager.cc:1398] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:28.765215 22847 consensus_queue.cc:260] T 37d7d0f4ab374b6fac1b537e54b030a6 P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.766151 22847 ts_tablet_manager.cc:1398] T 37d7d0f4ab374b6fac1b537e54b030a6 P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240503 08:11:28.769433 22847 tablet_bootstrap.cc:492] T 8aa3476fa91846358562fa4425410d16 P 6503743925434401a74aae303628e415: Bootstrap starting.
I20240503 08:11:28.770959 22846 tablet_bootstrap.cc:492] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:28.771265 22846 ts_tablet_manager.cc:1367] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20240503 08:11:28.772254 22847 tablet_bootstrap.cc:654] T 8aa3476fa91846358562fa4425410d16 P 6503743925434401a74aae303628e415: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:28.773548 22846 raft_consensus.cc:348] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.773531 22841 raft_consensus.cc:348] T 483431ac3a40425b90b051d72c5c3453 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:28.773767 22841 raft_consensus.cc:374] T 483431ac3a40425b90b051d72c5c3453 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:28.774055 22841 raft_consensus.cc:729] T 483431ac3a40425b90b051d72c5c3453 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8d441f0f6425b88ec78c8b0d7f3d5, State: Initialized, Role: FOLLOWER
I20240503 08:11:28.773896 22846 raft_consensus.cc:374] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:28.775177 22846 raft_consensus.cc:729] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:28.774998 22841 consensus_queue.cc:260] T 483431ac3a40425b90b051d72c5c3453 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:28.775548 22846 consensus_queue.cc:260] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.775754 22847 tablet_bootstrap.cc:492] T 8aa3476fa91846358562fa4425410d16 P 6503743925434401a74aae303628e415: No bootstrap required, opened a new log
I20240503 08:11:28.776028 22847 ts_tablet_manager.cc:1367] T 8aa3476fa91846358562fa4425410d16 P 6503743925434401a74aae303628e415: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20240503 08:11:28.776371 22846 ts_tablet_manager.cc:1398] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:11:28.776737 22846 tablet_bootstrap.cc:492] T 8aa3476fa91846358562fa4425410d16 P 3b6eea641d214c79988ec6ad8821a4bf: Bootstrap starting.
I20240503 08:11:28.777007 22843 tablet_bootstrap.cc:492] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:28.776963 22847 raft_consensus.cc:348] T 8aa3476fa91846358562fa4425410d16 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.777179 22843 ts_tablet_manager.cc:1367] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.019s
I20240503 08:11:28.777197 22847 raft_consensus.cc:374] T 8aa3476fa91846358562fa4425410d16 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:28.777269 22847 raft_consensus.cc:729] T 8aa3476fa91846358562fa4425410d16 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503743925434401a74aae303628e415, State: Initialized, Role: FOLLOWER
I20240503 08:11:28.777513 22847 consensus_queue.cc:260] T 8aa3476fa91846358562fa4425410d16 P 6503743925434401a74aae303628e415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.778048 22847 ts_tablet_manager.cc:1398] T 8aa3476fa91846358562fa4425410d16 P 6503743925434401a74aae303628e415: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:28.777963 22843 raft_consensus.cc:348] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.778191 22843 raft_consensus.cc:374] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:28.778296 22843 raft_consensus.cc:729] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:28.778543 22843 consensus_queue.cc:260] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.779090 22843 ts_tablet_manager.cc:1398] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:28.779361 22846 tablet_bootstrap.cc:654] T 8aa3476fa91846358562fa4425410d16 P 3b6eea641d214c79988ec6ad8821a4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:28.779404 22843 tablet_bootstrap.cc:492] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842: Bootstrap starting.
I20240503 08:11:28.782285 22846 tablet_bootstrap.cc:492] T 8aa3476fa91846358562fa4425410d16 P 3b6eea641d214c79988ec6ad8821a4bf: No bootstrap required, opened a new log
I20240503 08:11:28.782490 22846 ts_tablet_manager.cc:1367] T 8aa3476fa91846358562fa4425410d16 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:11:28.783540 22846 raft_consensus.cc:348] T 8aa3476fa91846358562fa4425410d16 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.783797 22846 raft_consensus.cc:374] T 8aa3476fa91846358562fa4425410d16 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:28.783936 22843 tablet_bootstrap.cc:654] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:28.783922 22846 raft_consensus.cc:729] T 8aa3476fa91846358562fa4425410d16 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eea641d214c79988ec6ad8821a4bf, State: Initialized, Role: FOLLOWER
I20240503 08:11:28.784248 22846 consensus_queue.cc:260] T 8aa3476fa91846358562fa4425410d16 P 3b6eea641d214c79988ec6ad8821a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.785295 22846 ts_tablet_manager.cc:1398] T 8aa3476fa91846358562fa4425410d16 P 3b6eea641d214c79988ec6ad8821a4bf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:28.785490 22841 ts_tablet_manager.cc:1398] T 483431ac3a40425b90b051d72c5c3453 P 74c8d441f0f6425b88ec78c8b0d7f3d5: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20240503 08:11:28.790148 22843 tablet_bootstrap.cc:492] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842: No bootstrap required, opened a new log
I20240503 08:11:28.790313 22843 ts_tablet_manager.cc:1367] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20240503 08:11:28.790916 22843 raft_consensus.cc:348] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.791091 22843 raft_consensus.cc:374] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:28.791164 22843 raft_consensus.cc:729] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Initialized, Role: FOLLOWER
I20240503 08:11:28.791431 22843 consensus_queue.cc:260] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.791903 22843 ts_tablet_manager.cc:1398] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:11:28.796667 22558 raft_consensus.cc:482] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:28.796900 22558 raft_consensus.cc:504] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.797538 22558 leader_election.cc:288] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:28.797959 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa3476fa91846358562fa4425410d16" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf" is_pre_election: true
I20240503 08:11:28.797981 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa3476fa91846358562fa4425410d16" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415" is_pre_election: true
I20240503 08:11:28.798171 21750 raft_consensus.cc:2454] T 8aa3476fa91846358562fa4425410d16 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 0.
I20240503 08:11:28.798192 21226 raft_consensus.cc:2454] T 8aa3476fa91846358562fa4425410d16 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 0.
I20240503 08:11:28.798576 21424 leader_election.cc:302] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e04594b1a5344d9bfefa4de05d70842, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:28.798882 22558 raft_consensus.cc:2781] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:28.798979 22558 raft_consensus.cc:482] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:28.799101 22558 raft_consensus.cc:3037] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:28.801499 22558 raft_consensus.cc:504] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.802114 22558 leader_election.cc:288] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:28.802549 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa3476fa91846358562fa4425410d16" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415"
I20240503 08:11:28.802529 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa3476fa91846358562fa4425410d16" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf"
I20240503 08:11:28.802740 21750 raft_consensus.cc:3037] T 8aa3476fa91846358562fa4425410d16 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:28.802784 21226 raft_consensus.cc:3037] T 8aa3476fa91846358562fa4425410d16 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:28.804666 21750 raft_consensus.cc:2454] T 8aa3476fa91846358562fa4425410d16 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:28.805141 21424 leader_election.cc:302] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e04594b1a5344d9bfefa4de05d70842, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:28.805217 21226 raft_consensus.cc:2454] T 8aa3476fa91846358562fa4425410d16 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:28.805475 22558 raft_consensus.cc:2781] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:28.805979 22558 raft_consensus.cc:686] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 LEADER]: Becoming Leader. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Running, Role: LEADER
I20240503 08:11:28.806279 22558 consensus_queue.cc:237] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.809648 20945 catalog_manager.cc:5484] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132). New cstate: current_term: 1 leader_uuid: "3e04594b1a5344d9bfefa4de05d70842" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:28.843204 22558 raft_consensus.cc:482] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:28.843415 22558 raft_consensus.cc:504] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.844743 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d7d0f4ab374b6fac1b537e54b030a6" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415" is_pre_election: true
I20240503 08:11:28.845034 21750 raft_consensus.cc:2454] T 37d7d0f4ab374b6fac1b537e54b030a6 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 0.
I20240503 08:11:28.845490 21424 leader_election.cc:302] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e04594b1a5344d9bfefa4de05d70842, 6503743925434401a74aae303628e415; no voters:
I20240503 08:11:28.845814 22558 leader_election.cc:288] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:28.845976 22558 raft_consensus.cc:2781] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:28.846086 22558 raft_consensus.cc:482] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:28.846184 22558 raft_consensus.cc:3037] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:28.846634 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d7d0f4ab374b6fac1b537e54b030a6" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf" is_pre_election: true
I20240503 08:11:28.846879 21226 raft_consensus.cc:2454] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 0.
I20240503 08:11:28.848625 22558 raft_consensus.cc:504] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.849213 22558 leader_election.cc:288] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525), 6503743925434401a74aae303628e415 (127.20.107.134:42791)
I20240503 08:11:28.849666 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d7d0f4ab374b6fac1b537e54b030a6" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503743925434401a74aae303628e415"
I20240503 08:11:28.849742 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d7d0f4ab374b6fac1b537e54b030a6" candidate_uuid: "3e04594b1a5344d9bfefa4de05d70842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6eea641d214c79988ec6ad8821a4bf"
I20240503 08:11:28.849876 21750 raft_consensus.cc:3037] T 37d7d0f4ab374b6fac1b537e54b030a6 P 6503743925434401a74aae303628e415 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:28.849951 21226 raft_consensus.cc:3037] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3b6eea641d214c79988ec6ad8821a4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:28.851897 21750 raft_consensus.cc:2454] T 37d7d0f4ab374b6fac1b537e54b030a6 P 6503743925434401a74aae303628e415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:28.851927 21226 raft_consensus.cc:2454] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3b6eea641d214c79988ec6ad8821a4bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e04594b1a5344d9bfefa4de05d70842 in term 1.
I20240503 08:11:28.852335 21423 leader_election.cc:302] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eea641d214c79988ec6ad8821a4bf, 3e04594b1a5344d9bfefa4de05d70842; no voters:
I20240503 08:11:28.852640 22558 raft_consensus.cc:2781] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:28.852787 22558 raft_consensus.cc:686] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [term 1 LEADER]: Becoming Leader. State: Replica: 3e04594b1a5344d9bfefa4de05d70842, State: Running, Role: LEADER
I20240503 08:11:28.853102 22558 consensus_queue.cc:237] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } }
I20240503 08:11:28.856333 20945 catalog_manager.cc:5484] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132). New cstate: current_term: 1 leader_uuid: "3e04594b1a5344d9bfefa4de05d70842" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e04594b1a5344d9bfefa4de05d70842" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 41603 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:29.011315 22686 raft_consensus.cc:482] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:29.011581 22686 raft_consensus.cc:504] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:29.012418 22686 leader_election.cc:288] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:29.013098 21618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483431ac3a40425b90b051d72c5c3453" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" is_pre_election: true
I20240503 08:11:29.013087 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483431ac3a40425b90b051d72c5c3453" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" is_pre_election: true
I20240503 08:11:29.013325 21618 raft_consensus.cc:2454] T 483431ac3a40425b90b051d72c5c3453 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 0.
I20240503 08:11:29.013351 21357 raft_consensus.cc:2454] T 483431ac3a40425b90b051d72c5c3453 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 0.
I20240503 08:11:29.013828 21027 leader_election.cc:302] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d9776f5d634ec2b167dc225659e4fd, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:29.014132 22686 raft_consensus.cc:2781] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:29.014276 22686 raft_consensus.cc:482] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:29.014390 22686 raft_consensus.cc:3037] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.016757 22686 raft_consensus.cc:504] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:29.017396 22686 leader_election.cc:288] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 election: Requested vote from peers 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537), 17af6d2a2fd14b6d93e232da6aa8f6a9 (127.20.107.133:43423)
I20240503 08:11:29.017715 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483431ac3a40425b90b051d72c5c3453" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5"
I20240503 08:11:29.017905 21357 raft_consensus.cc:3037] T 483431ac3a40425b90b051d72c5c3453 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.017927 21618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483431ac3a40425b90b051d72c5c3453" candidate_uuid: "18d9776f5d634ec2b167dc225659e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9"
I20240503 08:11:29.018132 21618 raft_consensus.cc:3037] T 483431ac3a40425b90b051d72c5c3453 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.019847 21357 raft_consensus.cc:2454] T 483431ac3a40425b90b051d72c5c3453 P 74c8d441f0f6425b88ec78c8b0d7f3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 1.
I20240503 08:11:29.020102 21618 raft_consensus.cc:2454] T 483431ac3a40425b90b051d72c5c3453 P 17af6d2a2fd14b6d93e232da6aa8f6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d9776f5d634ec2b167dc225659e4fd in term 1.
I20240503 08:11:29.020236 21027 leader_election.cc:302] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d9776f5d634ec2b167dc225659e4fd, 74c8d441f0f6425b88ec78c8b0d7f3d5; no voters:
I20240503 08:11:29.020539 22860 raft_consensus.cc:2781] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:29.020684 22860 raft_consensus.cc:686] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [term 1 LEADER]: Becoming Leader. State: Replica: 18d9776f5d634ec2b167dc225659e4fd, State: Running, Role: LEADER
I20240503 08:11:29.021009 22860 consensus_queue.cc:237] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } }
I20240503 08:11:29.024839 20945 catalog_manager.cc:5484] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129). New cstate: current_term: 1 leader_uuid: "18d9776f5d634ec2b167dc225659e4fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d9776f5d634ec2b167dc225659e4fd" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 39921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:29.201588 22558 consensus_queue.cc:1035] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:29.214473 22857 consensus_queue.cc:1035] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:29.317258 22558 consensus_queue.cc:1035] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503743925434401a74aae303628e415" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:29.331426 22857 consensus_queue.cc:1035] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6eea641d214c79988ec6ad8821a4bf" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:29.416002 22862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:29.416348 22862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:29.424536 22862 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240503 08:11:29.507241 22686 consensus_queue.cc:1035] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c8d441f0f6425b88ec78c8b0d7f3d5" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 37537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:29.531829 22686 consensus_queue.cc:1035] T 483431ac3a40425b90b051d72c5c3453 P 18d9776f5d634ec2b167dc225659e4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "17af6d2a2fd14b6d93e232da6aa8f6a9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:29.664441 21424 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.134:42791, user_credentials={real_user=slave}} blocked reactor thread for 44741.2us
INSERT report
rows total: 2000
time total: 267.241 ms
time per row: 0.13362 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:11:31.379715 21599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:31.403132 21206 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:31.406276 21073 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:31.464805 21729 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:31.466478 21468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:31.485280 21336 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:31.688262 21599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:31.695045 21206 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:31.711124 21468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:31.712940 21729 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:31.726167 21073 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:31.745725 21336 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:31.999645 21599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:32.003718 21073 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:32.014935 21206 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:32.019101 21468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:32.033105 21336 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:32.039855 21729 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:32.268762 21599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:32.302870 21206 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:32.313927 21073 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:32.343333 21468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:32.361253 21729 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:32.382645 21336 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:32.621155 21599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:32.628021 21073 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:32.646806 21206 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:32.668946 21468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:32.670424 21336 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:32.676664 21729 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:33.054842 21073 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:33.060775 21599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:33.067962 21729 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:33.069280 21206 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:33.082506 21468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:33.090217 21336 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
4bedc607804b4c579e99e7e8aea3f5b2 | 127.20.107.190:38313 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.107.148:43947 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
17af6d2a2fd14b6d93e232da6aa8f6a9 | 127.20.107.133:43423 | HEALTHY | /C | 5 | 0
18d9776f5d634ec2b167dc225659e4fd | 127.20.107.129:39921 | HEALTHY | /A | 6 | 0
3b6eea641d214c79988ec6ad8821a4bf | 127.20.107.130:35525 | HEALTHY | /A | 6 | 0
3e04594b1a5344d9bfefa4de05d70842 | 127.20.107.132:41603 | HEALTHY | /B | 8 | 0
6503743925434401a74aae303628e415 | 127.20.107.134:42791 | HEALTHY | /C | 4 | 0
74c8d441f0f6425b88ec78c8b0d7f3d5 | 127.20.107.131:37537 | HEALTHY | /B | 7 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/B | 2
/C | 2
/A | 2
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.20.107.129 | experimental | 127.20.107.129:39921
local_ip_for_outbound_sockets | 127.20.107.130 | experimental | 127.20.107.130:35525
local_ip_for_outbound_sockets | 127.20.107.131 | experimental | 127.20.107.131:37537
local_ip_for_outbound_sockets | 127.20.107.132 | experimental | 127.20.107.132:41603
local_ip_for_outbound_sockets | 127.20.107.133 | experimental | 127.20.107.133:43423
local_ip_for_outbound_sockets | 127.20.107.134 | experimental | 127.20.107.134:42791
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.20.107.129:39921
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.20.107.130:35525
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.20.107.131:37537
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.20.107.132:41603
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.20.107.133:43423
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.20.107.134:42791
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.107.148:43947 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_10 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_11 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 18
Median | 18
Third Quartile | 18
Maximum | 18
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 12
Tablets | 36
Replicas | 108
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:11:33.176849 20910 log_verifier.cc:126] Checking tablet 08c639a95ae940808c97c5dfcdc13caf
I20240503 08:11:33.217629 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 08c639a95ae940808c97c5dfcdc13caf
I20240503 08:11:33.217823 20910 log_verifier.cc:126] Checking tablet 0c52927816c44a9a87a5fa95ada7801a
I20240503 08:11:33.253608 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0c52927816c44a9a87a5fa95ada7801a
I20240503 08:11:33.253747 20910 log_verifier.cc:126] Checking tablet 0e049e82d55547288f5911a11a47ffb6
I20240503 08:11:33.290416 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0e049e82d55547288f5911a11a47ffb6
I20240503 08:11:33.290560 20910 log_verifier.cc:126] Checking tablet 1b3dfcb5d6c943bea49ebfb7ebb50796
I20240503 08:11:33.335065 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1b3dfcb5d6c943bea49ebfb7ebb50796
I20240503 08:11:33.335263 20910 log_verifier.cc:126] Checking tablet 1d9dc83cb659489a82b0fff6694cb1a7
I20240503 08:11:33.385140 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1d9dc83cb659489a82b0fff6694cb1a7
I20240503 08:11:33.385265 20910 log_verifier.cc:126] Checking tablet 28305e23ecff40e9bce63ccf908cecb5
I20240503 08:11:33.425973 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 28305e23ecff40e9bce63ccf908cecb5
I20240503 08:11:33.426100 20910 log_verifier.cc:126] Checking tablet 2b12dd2cc18748d2a3251f6ad7e6e482
I20240503 08:11:33.468535 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2b12dd2cc18748d2a3251f6ad7e6e482
I20240503 08:11:33.468652 20910 log_verifier.cc:126] Checking tablet 2b4d3741c904464dba15f0659614d1f8
I20240503 08:11:33.508486 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2b4d3741c904464dba15f0659614d1f8
I20240503 08:11:33.508595 20910 log_verifier.cc:126] Checking tablet 2c5baa3ecf264275b6006bcf0fef7ca9
I20240503 08:11:33.549458 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2c5baa3ecf264275b6006bcf0fef7ca9
I20240503 08:11:33.549566 20910 log_verifier.cc:126] Checking tablet 37d7d0f4ab374b6fac1b537e54b030a6
I20240503 08:11:33.590229 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 37d7d0f4ab374b6fac1b537e54b030a6
I20240503 08:11:33.590348 20910 log_verifier.cc:126] Checking tablet 3818719d4dc64f60b25efdc49c7e602b
I20240503 08:11:33.631950 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3818719d4dc64f60b25efdc49c7e602b
I20240503 08:11:33.632069 20910 log_verifier.cc:126] Checking tablet 483431ac3a40425b90b051d72c5c3453
I20240503 08:11:33.676034 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 483431ac3a40425b90b051d72c5c3453
I20240503 08:11:33.676167 20910 log_verifier.cc:126] Checking tablet 4b5c3da0019f48dd8592eb5b5d393256
I20240503 08:11:33.718772 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4b5c3da0019f48dd8592eb5b5d393256
I20240503 08:11:33.718880 20910 log_verifier.cc:126] Checking tablet 4d5a2cab7232499bb4aba2c6d8713723
I20240503 08:11:33.758901 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4d5a2cab7232499bb4aba2c6d8713723
I20240503 08:11:33.759032 20910 log_verifier.cc:126] Checking tablet 4e77b4ff1be549068fe4ebb4de6d46a1
I20240503 08:11:33.801362 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4e77b4ff1be549068fe4ebb4de6d46a1
I20240503 08:11:33.801470 20910 log_verifier.cc:126] Checking tablet 533a5cf7eba34035a066a47862b54445
I20240503 08:11:33.843860 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 533a5cf7eba34035a066a47862b54445
I20240503 08:11:33.843969 20910 log_verifier.cc:126] Checking tablet 5ac07d68002548958c4eeede92fcd4be
I20240503 08:11:33.885810 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5ac07d68002548958c4eeede92fcd4be
I20240503 08:11:33.885919 20910 log_verifier.cc:126] Checking tablet 5fe5f486eb394f12b698cb506e51ab3b
I20240503 08:11:33.926295 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5fe5f486eb394f12b698cb506e51ab3b
I20240503 08:11:33.926402 20910 log_verifier.cc:126] Checking tablet 64a39d113638472580a5932bcc8531a3
I20240503 08:11:33.967417 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 64a39d113638472580a5932bcc8531a3
I20240503 08:11:33.967576 20910 log_verifier.cc:126] Checking tablet 6d23a8d6a1a2476b82d441f5d289b9fb
I20240503 08:11:34.014911 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6d23a8d6a1a2476b82d441f5d289b9fb
I20240503 08:11:34.015051 20910 log_verifier.cc:126] Checking tablet 7f0614c8c6ce4539925785c065829ad2
I20240503 08:11:34.060060 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7f0614c8c6ce4539925785c065829ad2
I20240503 08:11:34.060199 20910 log_verifier.cc:126] Checking tablet 87221edea55a4bcc85e1a0d9bf8f0c55
I20240503 08:11:34.102695 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 87221edea55a4bcc85e1a0d9bf8f0c55
I20240503 08:11:34.102825 20910 log_verifier.cc:126] Checking tablet 8aa3476fa91846358562fa4425410d16
I20240503 08:11:34.141602 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8aa3476fa91846358562fa4425410d16
I20240503 08:11:34.141721 20910 log_verifier.cc:126] Checking tablet 8c221fe52b8a4ce083bd4ecb3f906dc0
I20240503 08:11:34.181790 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8c221fe52b8a4ce083bd4ecb3f906dc0
I20240503 08:11:34.181911 20910 log_verifier.cc:126] Checking tablet 95f72713a7634db0a84ba8c144148db5
I20240503 08:11:34.220978 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 95f72713a7634db0a84ba8c144148db5
I20240503 08:11:34.221117 20910 log_verifier.cc:126] Checking tablet 977a045f5f4b469f846aad8acf86b0b0
I20240503 08:11:34.262303 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 977a045f5f4b469f846aad8acf86b0b0
I20240503 08:11:34.262439 20910 log_verifier.cc:126] Checking tablet 97fd5f8f822f49b49f18cad0989c1c7e
I20240503 08:11:34.308357 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 97fd5f8f822f49b49f18cad0989c1c7e
I20240503 08:11:34.308468 20910 log_verifier.cc:126] Checking tablet 9fc3222fc2a54a2087b347efd3b18f0a
I20240503 08:11:34.350210 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9fc3222fc2a54a2087b347efd3b18f0a
I20240503 08:11:34.350335 20910 log_verifier.cc:126] Checking tablet a838cc18e64b4fd0be47f3c9625a209d
I20240503 08:11:34.393523 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a838cc18e64b4fd0be47f3c9625a209d
I20240503 08:11:34.393647 20910 log_verifier.cc:126] Checking tablet a9fdcbd5803e49668916fc8b4d06dbbf
I20240503 08:11:34.434893 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a9fdcbd5803e49668916fc8b4d06dbbf
I20240503 08:11:34.435035 20910 log_verifier.cc:126] Checking tablet be04a137b2074903a273b3020563c5c3
I20240503 08:11:34.477761 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet be04a137b2074903a273b3020563c5c3
I20240503 08:11:34.477900 20910 log_verifier.cc:126] Checking tablet caf143a06fd3435fa538f1550bce641f
I20240503 08:11:34.518503 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet caf143a06fd3435fa538f1550bce641f
I20240503 08:11:34.518640 20910 log_verifier.cc:126] Checking tablet d192acaf3cf74b508944391d49c95cbc
I20240503 08:11:34.560739 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d192acaf3cf74b508944391d49c95cbc
I20240503 08:11:34.560884 20910 log_verifier.cc:126] Checking tablet d7c9e2406d734bb68b7732190dbdd535
I20240503 08:11:34.602813 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d7c9e2406d734bb68b7732190dbdd535
I20240503 08:11:34.602944 20910 log_verifier.cc:126] Checking tablet e5cdd453d491408095e460eb78265e03
I20240503 08:11:34.642462 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e5cdd453d491408095e460eb78265e03
I20240503 08:11:34.642587 20910 log_verifier.cc:126] Checking tablet fbf2ad4b08ec4646896e6add57ca3caa
I20240503 08:11:34.684190 20910 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fbf2ad4b08ec4646896e6add57ca3caa
I20240503 08:11:34.704026 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 21012
I20240503 08:11:34.710871 21133 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:34.846096 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 21012
W20240503 08:11:34.871752 21553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.107.129:39921 (error 108)
W20240503 08:11:34.871774 21291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.107.129:39921 (error 108)
W20240503 08:11:34.871917 21293 connection.cc:535] server connection from 127.20.107.129:48509 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:11:34.872495 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 21143
W20240503 08:11:34.875119 21553 consensus_peers.cc:485] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 -> Peer 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921): Couldn't send request to peer 18d9776f5d634ec2b167dc225659e4fd. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.129:39921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:34.875560 21291 consensus_peers.cc:485] T 977a045f5f4b469f846aad8acf86b0b0 P 74c8d441f0f6425b88ec78c8b0d7f3d5 -> Peer 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921): Couldn't send request to peer 18d9776f5d634ec2b167dc225659e4fd. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.129:39921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:34.879228 21265 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:11:34.895861 21553 consensus_peers.cc:485] T 5fe5f486eb394f12b698cb506e51ab3b P 17af6d2a2fd14b6d93e232da6aa8f6a9 -> Peer 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921): Couldn't send request to peer 18d9776f5d634ec2b167dc225659e4fd. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.129:39921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:34.975162 21553 consensus_peers.cc:485] T a9fdcbd5803e49668916fc8b4d06dbbf P 17af6d2a2fd14b6d93e232da6aa8f6a9 -> Peer 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921): Couldn't send request to peer 18d9776f5d634ec2b167dc225659e4fd. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.129:39921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:34.975926 21553 consensus_peers.cc:485] T 08c639a95ae940808c97c5dfcdc13caf P 17af6d2a2fd14b6d93e232da6aa8f6a9 -> Peer 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921): Couldn't send request to peer 18d9776f5d634ec2b167dc225659e4fd. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.129:39921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:34.980095 21291 consensus_peers.cc:485] T 5ac07d68002548958c4eeede92fcd4be P 74c8d441f0f6425b88ec78c8b0d7f3d5 -> Peer 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921): Couldn't send request to peer 18d9776f5d634ec2b167dc225659e4fd. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.129:39921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:35.019610 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 21143
W20240503 08:11:35.036372 21291 consensus_peers.cc:485] T 533a5cf7eba34035a066a47862b54445 P 74c8d441f0f6425b88ec78c8b0d7f3d5 -> Peer 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921): Couldn't send request to peer 18d9776f5d634ec2b167dc225659e4fd. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.129:39921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:35.040388 21291 consensus_peers.cc:485] T a838cc18e64b4fd0be47f3c9625a209d P 74c8d441f0f6425b88ec78c8b0d7f3d5 -> Peer 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921): Couldn't send request to peer 18d9776f5d634ec2b167dc225659e4fd. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.129:39921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:35.043843 21422 connection.cc:535] server connection from 127.20.107.130:51067 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:11:35.043890 21685 connection.cc:535] server connection from 127.20.107.130:36679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:11:35.044111 21685 connection.cc:535] client connection to 127.20.107.130:35525 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:11:35.044270 21423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.107.130:35525 (error 108)
I20240503 08:11:35.044392 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 21274
W20240503 08:11:35.044339 21685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:11:35.047641 21685 consensus_peers.cc:485] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 -> Peer 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525): Couldn't send request to peer 3b6eea641d214c79988ec6ad8821a4bf. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.130:35525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:35.047932 21423 consensus_peers.cc:485] T 1b3dfcb5d6c943bea49ebfb7ebb50796 P 3e04594b1a5344d9bfefa4de05d70842 -> Peer 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525): Couldn't send request to peer 3b6eea641d214c79988ec6ad8821a4bf. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.130:35525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:35.048035 21423 consensus_peers.cc:485] T 37d7d0f4ab374b6fac1b537e54b030a6 P 3e04594b1a5344d9bfefa4de05d70842 -> Peer 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525): Couldn't send request to peer 3b6eea641d214c79988ec6ad8821a4bf. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.130:35525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:35.051760 21396 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:11:35.064065 21685 consensus_peers.cc:485] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 -> Peer 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525): Couldn't send request to peer 3b6eea641d214c79988ec6ad8821a4bf. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.130:35525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:35.064093 21423 consensus_peers.cc:485] T be04a137b2074903a273b3020563c5c3 P 3e04594b1a5344d9bfefa4de05d70842 -> Peer 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525): Couldn't send request to peer 3b6eea641d214c79988ec6ad8821a4bf. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.130:35525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:35.153122 21423 consensus_peers.cc:485] T 8aa3476fa91846358562fa4425410d16 P 3e04594b1a5344d9bfefa4de05d70842 -> Peer 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525): Couldn't send request to peer 3b6eea641d214c79988ec6ad8821a4bf. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.130:35525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:35.159448 21685 consensus_peers.cc:485] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 -> Peer 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525): Couldn't send request to peer 3b6eea641d214c79988ec6ad8821a4bf. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.130:35525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:35.185724 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 21274
W20240503 08:11:35.189363 21423 consensus_peers.cc:485] T 2b4d3741c904464dba15f0659614d1f8 P 3e04594b1a5344d9bfefa4de05d70842 -> Peer 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525): Couldn't send request to peer 3b6eea641d214c79988ec6ad8821a4bf. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.130:35525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:35.210856 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 21405
W20240503 08:11:35.211622 21552 consensus_peers.cc:485] T e5cdd453d491408095e460eb78265e03 P 17af6d2a2fd14b6d93e232da6aa8f6a9 -> Peer 74c8d441f0f6425b88ec78c8b0d7f3d5 (127.20.107.131:37537): Couldn't send request to peer 74c8d441f0f6425b88ec78c8b0d7f3d5. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.131:37537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:35.217221 21527 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:11:35.274163 21553 consensus_peers.cc:485] T 2c5baa3ecf264275b6006bcf0fef7ca9 P 17af6d2a2fd14b6d93e232da6aa8f6a9 -> Peer 18d9776f5d634ec2b167dc225659e4fd (127.20.107.129:39921): Couldn't send request to peer 18d9776f5d634ec2b167dc225659e4fd. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.129:39921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:35.354200 21423 consensus_peers.cc:485] T fbf2ad4b08ec4646896e6add57ca3caa P 3e04594b1a5344d9bfefa4de05d70842 -> Peer 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525): Couldn't send request to peer 3b6eea641d214c79988ec6ad8821a4bf. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.130:35525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:35.354380 21423 consensus_peers.cc:485] T 28305e23ecff40e9bce63ccf908cecb5 P 3e04594b1a5344d9bfefa4de05d70842 -> Peer 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525): Couldn't send request to peer 3b6eea641d214c79988ec6ad8821a4bf. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.130:35525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:35.354849 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 21405
I20240503 08:11:35.380995 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 21536
I20240503 08:11:35.387182 21659 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:11:35.413780 21685 consensus_peers.cc:485] T 0e049e82d55547288f5911a11a47ffb6 P 6503743925434401a74aae303628e415 -> Peer 3b6eea641d214c79988ec6ad8821a4bf (127.20.107.130:35525): Couldn't send request to peer 3b6eea641d214c79988ec6ad8821a4bf. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.130:35525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:35.475283 21684 consensus_peers.cc:485] T 95f72713a7634db0a84ba8c144148db5 P 6503743925434401a74aae303628e415 -> Peer 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603): Couldn't send request to peer 3e04594b1a5344d9bfefa4de05d70842. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.132:41603: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:35.520706 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 21536
W20240503 08:11:35.524502 21684 consensus_peers.cc:485] T 4d5a2cab7232499bb4aba2c6d8713723 P 6503743925434401a74aae303628e415 -> Peer 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603): Couldn't send request to peer 3e04594b1a5344d9bfefa4de05d70842. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.132:41603: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:35.545729 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 21667
W20240503 08:11:35.546665 21684 consensus_peers.cc:485] T 8c221fe52b8a4ce083bd4ecb3f906dc0 P 6503743925434401a74aae303628e415 -> Peer 3e04594b1a5344d9bfefa4de05d70842 (127.20.107.132:41603): Couldn't send request to peer 3e04594b1a5344d9bfefa4de05d70842. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.132:41603: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:35.551565 21789 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:35.681545 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 21667
I20240503 08:11:35.705178 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 20922
I20240503 08:11:35.706175 20983 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:35.856397 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 20922
2024-05-03T08:11:35Z chronyd exiting
[ OK ] LocationAwareRebalancingBasicTest.Basic (29251 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (29251 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240503 08:11:36.011526 20910 test_util.cc:268] Using random seed: 1152756447
I20240503 08:11:36.013365 20910 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:36.013440 20910 ts_itest-base.cc:113] --------------
I20240503 08:11:36.013491 20910 ts_itest-base.cc:114] 5 tablet servers
I20240503 08:11:36.013567 20910 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:11:36.013687 20910 ts_itest-base.cc:116] --------------
2024-05-03T08:11:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:36Z Disabled control of system clock
I20240503 08:11:36.050055 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.107.190:34385
--webserver_interface=127.20.107.190
--webserver_port=0
--builtin_ntp_servers=127.20.107.148:44261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.107.190:34385
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 08:11:36.391217 23247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:36.391566 23247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:36.391685 23247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:36.399485 23247 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:36.399623 23247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:11:36.399724 23247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:36.399808 23247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:11:36.399866 23247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:11:36.409199 23247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:44261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.107.190:34385
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.107.190:34385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.107.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:36.410662 23247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:36.412557 23247 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:36.423868 23255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:36.424571 23252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:36.425443 23253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:36.426259 23247 server_base.cc:1034] running on GCE node
I20240503 08:11:36.427021 23247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:36.428452 23247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:36.429648 23247 hybrid_clock.cc:648] HybridClock initialized: now 1714723896429600 us; error 40 us; skew 500 ppm
I20240503 08:11:36.429981 23247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:36.436084 23247 webserver.cc:460] Webserver started at http://127.20.107.190:46381/ using document root <none> and password file <none>
I20240503 08:11:36.436826 23247 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:36.436988 23247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:36.437299 23247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:36.439888 23247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f09eb88e3dcd41fb8cfe350659c84fd4"
format_stamp: "Formatted at 2024-05-03 08:11:36 on dist-test-slave-4tl1"
I20240503 08:11:36.440636 23247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f09eb88e3dcd41fb8cfe350659c84fd4"
format_stamp: "Formatted at 2024-05-03 08:11:36 on dist-test-slave-4tl1"
I20240503 08:11:36.445910 23247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240503 08:11:36.449471 23261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:36.450868 23247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:36.451030 23247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f09eb88e3dcd41fb8cfe350659c84fd4"
format_stamp: "Formatted at 2024-05-03 08:11:36 on dist-test-slave-4tl1"
I20240503 08:11:36.451225 23247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:36.478037 23247 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:36.478899 23247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:36.479152 23247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:36.497905 23247 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.190:34385
I20240503 08:11:36.497923 23312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.190:34385 every 8 connection(s)
I20240503 08:11:36.499359 23247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:11:36.503191 23313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.505440 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 23247
I20240503 08:11:36.505672 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:11:36.514856 23313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4: Bootstrap starting.
I20240503 08:11:36.518731 23313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.520016 23313 log.cc:824] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:36.522708 23313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4: No bootstrap required, opened a new log
I20240503 08:11:36.528800 23313 raft_consensus.cc:348] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09eb88e3dcd41fb8cfe350659c84fd4" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 34385 } }
I20240503 08:11:36.529167 23313 raft_consensus.cc:374] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.529270 23313 raft_consensus.cc:729] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09eb88e3dcd41fb8cfe350659c84fd4, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.529888 23313 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09eb88e3dcd41fb8cfe350659c84fd4" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 34385 } }
I20240503 08:11:36.530083 23313 raft_consensus.cc:388] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:36.530205 23313 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:36.530354 23313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.532604 23313 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09eb88e3dcd41fb8cfe350659c84fd4" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 34385 } }
I20240503 08:11:36.533185 23313 leader_election.cc:302] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f09eb88e3dcd41fb8cfe350659c84fd4; no voters:
I20240503 08:11:36.533794 23313 leader_election.cc:288] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:36.533919 23318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:36.534596 23318 raft_consensus.cc:686] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [term 1 LEADER]: Becoming Leader. State: Replica: f09eb88e3dcd41fb8cfe350659c84fd4, State: Running, Role: LEADER
I20240503 08:11:36.535136 23318 consensus_queue.cc:237] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09eb88e3dcd41fb8cfe350659c84fd4" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 34385 } }
I20240503 08:11:36.535823 23313 sys_catalog.cc:564] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:36.537870 23320 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09eb88e3dcd41fb8cfe350659c84fd4. Latest consensus state: current_term: 1 leader_uuid: "f09eb88e3dcd41fb8cfe350659c84fd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09eb88e3dcd41fb8cfe350659c84fd4" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 34385 } } }
I20240503 08:11:36.538277 23320 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:36.538681 23319 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f09eb88e3dcd41fb8cfe350659c84fd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09eb88e3dcd41fb8cfe350659c84fd4" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 34385 } } }
I20240503 08:11:36.539073 23319 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:36.540560 23323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:36.546586 23323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:36.555610 23323 catalog_manager.cc:1347] Generated new cluster ID: 2fe5a447cd5d441eb65b57ad70d54c22
I20240503 08:11:36.555755 23323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:36.572353 23323 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:36.573611 23323 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:36.581274 23323 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f09eb88e3dcd41fb8cfe350659c84fd4: Generated new TSK 0
I20240503 08:11:36.582448 23323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:36.589001 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.129:0
--local_ip_for_outbound_sockets=127.20.107.129
--webserver_interface=127.20.107.129
--webserver_port=0
--tserver_master_addrs=127.20.107.190:34385
--builtin_ntp_servers=127.20.107.148:44261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:11:36.935410 23337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:36.935747 23337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:36.935873 23337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:36.943833 23337 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:36.943979 23337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:36.944093 23337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.129
I20240503 08:11:36.953519 23337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:44261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.107.129
--webserver_port=0
--tserver_master_addrs=127.20.107.190:34385
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.129
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:36.954841 23337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:36.956622 23337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:36.969539 23342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:36.970475 23343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:36.971077 23345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:36.972039 23337 server_base.cc:1034] running on GCE node
I20240503 08:11:36.972704 23337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:36.973948 23337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:36.975174 23337 hybrid_clock.cc:648] HybridClock initialized: now 1714723896975125 us; error 50 us; skew 500 ppm
I20240503 08:11:36.975513 23337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:36.977885 23337 webserver.cc:460] Webserver started at http://127.20.107.129:40745/ using document root <none> and password file <none>
I20240503 08:11:36.978606 23337 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:36.978731 23337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:36.979043 23337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:36.981678 23337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1141dfda05b543e484c0bc5065da7f3e"
format_stamp: "Formatted at 2024-05-03 08:11:36 on dist-test-slave-4tl1"
I20240503 08:11:36.982400 23337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1141dfda05b543e484c0bc5065da7f3e"
format_stamp: "Formatted at 2024-05-03 08:11:36 on dist-test-slave-4tl1"
I20240503 08:11:36.987505 23337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:11:36.990878 23351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:36.992215 23337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:36.992380 23337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1141dfda05b543e484c0bc5065da7f3e"
format_stamp: "Formatted at 2024-05-03 08:11:36 on dist-test-slave-4tl1"
I20240503 08:11:36.992570 23337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:37.007189 23337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:37.008060 23337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:37.008334 23337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:37.009326 23337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:37.010798 23337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:37.010896 23337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.011013 23337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:37.011086 23337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.043008 23337 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.129:44625
I20240503 08:11:37.043064 23463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.129:44625 every 8 connection(s)
I20240503 08:11:37.044320 23337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:37.044638 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 23337
I20240503 08:11:37.044829 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:11:37.048552 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.130:0
--local_ip_for_outbound_sockets=127.20.107.130
--webserver_interface=127.20.107.130
--webserver_port=0
--tserver_master_addrs=127.20.107.190:34385
--builtin_ntp_servers=127.20.107.148:44261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:37.058319 23464 heartbeater.cc:347] Connected to a master server at 127.20.107.190:34385
I20240503 08:11:37.058652 23464 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:37.059547 23464 heartbeater.cc:511] Master 127.20.107.190:34385 requested a full tablet report, sending...
I20240503 08:11:37.061991 23278 ts_manager.cc:194] Registered new tserver with Master: 1141dfda05b543e484c0bc5065da7f3e (127.20.107.129:44625)
I20240503 08:11:37.064328 23278 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.129:44631
W20240503 08:11:37.382755 23468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:37.383056 23468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:37.383200 23468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:37.390568 23468 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:37.390718 23468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:37.390834 23468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.130
I20240503 08:11:37.399953 23468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:44261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.107.130
--webserver_port=0
--tserver_master_addrs=127.20.107.190:34385
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.130
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:37.401279 23468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:37.402958 23468 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:37.415693 23473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.416625 23474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.417377 23476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:37.418423 23468 server_base.cc:1034] running on GCE node
I20240503 08:11:37.419046 23468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:37.420060 23468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:37.421257 23468 hybrid_clock.cc:648] HybridClock initialized: now 1714723897421209 us; error 39 us; skew 500 ppm
I20240503 08:11:37.421583 23468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:37.423830 23468 webserver.cc:460] Webserver started at http://127.20.107.130:41509/ using document root <none> and password file <none>
I20240503 08:11:37.424564 23468 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:37.424669 23468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:37.425017 23468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:37.427632 23468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d8d4bc8fff4042b69580a73e50eb743e"
format_stamp: "Formatted at 2024-05-03 08:11:37 on dist-test-slave-4tl1"
I20240503 08:11:37.428376 23468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d8d4bc8fff4042b69580a73e50eb743e"
format_stamp: "Formatted at 2024-05-03 08:11:37 on dist-test-slave-4tl1"
I20240503 08:11:37.433773 23468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 08:11:37.437215 23482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.438545 23468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240503 08:11:37.438724 23468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d8d4bc8fff4042b69580a73e50eb743e"
format_stamp: "Formatted at 2024-05-03 08:11:37 on dist-test-slave-4tl1"
I20240503 08:11:37.438918 23468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:37.470492 23468 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:37.471465 23468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:37.471846 23468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:37.472903 23468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:37.474449 23468 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:37.474552 23468 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.474680 23468 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:37.474759 23468 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.507647 23468 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.130:34923
I20240503 08:11:37.507715 23594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.130:34923 every 8 connection(s)
I20240503 08:11:37.509131 23468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:11:37.516033 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 23468
I20240503 08:11:37.516265 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:11:37.519515 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.131:0
--local_ip_for_outbound_sockets=127.20.107.131
--webserver_interface=127.20.107.131
--webserver_port=0
--tserver_master_addrs=127.20.107.190:34385
--builtin_ntp_servers=127.20.107.148:44261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:37.523299 23595 heartbeater.cc:347] Connected to a master server at 127.20.107.190:34385
I20240503 08:11:37.523603 23595 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:37.524442 23595 heartbeater.cc:511] Master 127.20.107.190:34385 requested a full tablet report, sending...
I20240503 08:11:37.525952 23278 ts_manager.cc:194] Registered new tserver with Master: d8d4bc8fff4042b69580a73e50eb743e (127.20.107.130:34923)
I20240503 08:11:37.526818 23278 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.130:47119
W20240503 08:11:37.856236 23598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:37.856549 23598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:37.856700 23598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:37.864410 23598 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:37.864575 23598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:37.864732 23598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.131
I20240503 08:11:37.874044 23598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:44261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.107.131
--webserver_port=0
--tserver_master_addrs=127.20.107.190:34385
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.131
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:37.875425 23598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:37.877231 23598 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:37.889535 23604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.890326 23606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.891024 23603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:37.892024 23598 server_base.cc:1034] running on GCE node
I20240503 08:11:37.892673 23598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:37.901306 23598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:37.902591 23598 hybrid_clock.cc:648] HybridClock initialized: now 1714723897902525 us; error 68 us; skew 500 ppm
I20240503 08:11:37.902983 23598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:37.905412 23598 webserver.cc:460] Webserver started at http://127.20.107.131:36237/ using document root <none> and password file <none>
I20240503 08:11:37.906101 23598 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:37.906195 23598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:37.906469 23598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:37.909245 23598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f05cb769503641fcaf7fc8c2e9250b1a"
format_stamp: "Formatted at 2024-05-03 08:11:37 on dist-test-slave-4tl1"
I20240503 08:11:37.909940 23598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f05cb769503641fcaf7fc8c2e9250b1a"
format_stamp: "Formatted at 2024-05-03 08:11:37 on dist-test-slave-4tl1"
I20240503 08:11:37.914924 23598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:11:37.918090 23612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.919312 23598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:37.919437 23598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f05cb769503641fcaf7fc8c2e9250b1a"
format_stamp: "Formatted at 2024-05-03 08:11:37 on dist-test-slave-4tl1"
I20240503 08:11:37.919593 23598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:37.932934 23598 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:37.933701 23598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:37.933956 23598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:37.935058 23598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:37.936528 23598 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:37.936618 23598 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.936741 23598 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:37.936803 23598 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.965096 23598 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.131:35111
I20240503 08:11:37.965159 23724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.131:35111 every 8 connection(s)
I20240503 08:11:37.966588 23598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:37.974375 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 23598
I20240503 08:11:37.974542 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:11:37.977972 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.132:0
--local_ip_for_outbound_sockets=127.20.107.132
--webserver_interface=127.20.107.132
--webserver_port=0
--tserver_master_addrs=127.20.107.190:34385
--builtin_ntp_servers=127.20.107.148:44261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:37.979851 23725 heartbeater.cc:347] Connected to a master server at 127.20.107.190:34385
I20240503 08:11:37.980160 23725 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:37.981045 23725 heartbeater.cc:511] Master 127.20.107.190:34385 requested a full tablet report, sending...
I20240503 08:11:37.982374 23278 ts_manager.cc:194] Registered new tserver with Master: f05cb769503641fcaf7fc8c2e9250b1a (127.20.107.131:35111)
I20240503 08:11:37.983131 23278 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.131:35885
I20240503 08:11:38.067564 23464 heartbeater.cc:503] Master 127.20.107.190:34385 was elected leader, sending a full tablet report...
W20240503 08:11:38.303906 23728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:38.304222 23728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:38.304340 23728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:38.311741 23728 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:38.311893 23728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:38.312045 23728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.132
I20240503 08:11:38.320950 23728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:44261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.107.132
--webserver_port=0
--tserver_master_addrs=127.20.107.190:34385
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.132
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:38.322363 23728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:38.324214 23728 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:38.336273 23734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:38.336833 23733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:38.337749 23736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:38.338528 23728 server_base.cc:1034] running on GCE node
I20240503 08:11:38.339104 23728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:38.339987 23728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:38.341168 23728 hybrid_clock.cc:648] HybridClock initialized: now 1714723898341106 us; error 53 us; skew 500 ppm
I20240503 08:11:38.341451 23728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:38.343734 23728 webserver.cc:460] Webserver started at http://127.20.107.132:43211/ using document root <none> and password file <none>
I20240503 08:11:38.344442 23728 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:38.344539 23728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:38.344820 23728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:38.347362 23728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1"
format_stamp: "Formatted at 2024-05-03 08:11:38 on dist-test-slave-4tl1"
I20240503 08:11:38.348058 23728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1"
format_stamp: "Formatted at 2024-05-03 08:11:38 on dist-test-slave-4tl1"
I20240503 08:11:38.353109 23728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 08:11:38.356453 23742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:38.357888 23728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:38.358032 23728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1"
format_stamp: "Formatted at 2024-05-03 08:11:38 on dist-test-slave-4tl1"
I20240503 08:11:38.358217 23728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:38.383834 23728 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:38.384670 23728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:38.384971 23728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:38.385947 23728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:38.387374 23728 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:38.387460 23728 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:38.387534 23728 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:38.387588 23728 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:38.418835 23728 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.132:38361
I20240503 08:11:38.418856 23854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.132:38361 every 8 connection(s)
I20240503 08:11:38.420246 23728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:38.430235 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 23728
I20240503 08:11:38.430454 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:11:38.433406 23855 heartbeater.cc:347] Connected to a master server at 127.20.107.190:34385
I20240503 08:11:38.433671 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.133:0
--local_ip_for_outbound_sockets=127.20.107.133
--webserver_interface=127.20.107.133
--webserver_port=0
--tserver_master_addrs=127.20.107.190:34385
--builtin_ntp_servers=127.20.107.148:44261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:38.433713 23855 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:38.434576 23855 heartbeater.cc:511] Master 127.20.107.190:34385 requested a full tablet report, sending...
I20240503 08:11:38.436182 23278 ts_manager.cc:194] Registered new tserver with Master: e517a0e8f9ac44daaa6e9be4cc47a2c1 (127.20.107.132:38361)
I20240503 08:11:38.437361 23278 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.132:33311
I20240503 08:11:38.529850 23595 heartbeater.cc:503] Master 127.20.107.190:34385 was elected leader, sending a full tablet report...
W20240503 08:11:38.767711 23858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:38.768008 23858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:38.768096 23858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:38.775810 23858 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:38.775940 23858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:38.776082 23858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.133
I20240503 08:11:38.785246 23858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:44261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.107.133
--webserver_port=0
--tserver_master_addrs=127.20.107.190:34385
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.133
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:38.786602 23858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:38.788275 23858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:38.800678 23863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:38.801285 23864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:38.802170 23866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:38.802927 23858 server_base.cc:1034] running on GCE node
I20240503 08:11:38.803496 23858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:38.804440 23858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:38.805668 23858 hybrid_clock.cc:648] HybridClock initialized: now 1714723898805600 us; error 50 us; skew 500 ppm
I20240503 08:11:38.806026 23858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:38.808266 23858 webserver.cc:460] Webserver started at http://127.20.107.133:34933/ using document root <none> and password file <none>
I20240503 08:11:38.809034 23858 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:38.809154 23858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:38.809465 23858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:38.812203 23858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "44375967cba843c1ae87fa471159b6c2"
format_stamp: "Formatted at 2024-05-03 08:11:38 on dist-test-slave-4tl1"
I20240503 08:11:38.812996 23858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "44375967cba843c1ae87fa471159b6c2"
format_stamp: "Formatted at 2024-05-03 08:11:38 on dist-test-slave-4tl1"
I20240503 08:11:38.817999 23858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20240503 08:11:38.821264 23872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:38.822561 23858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240503 08:11:38.822778 23858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "44375967cba843c1ae87fa471159b6c2"
format_stamp: "Formatted at 2024-05-03 08:11:38 on dist-test-slave-4tl1"
I20240503 08:11:38.823022 23858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:38.849184 23858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:38.850088 23858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:38.850355 23858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:38.851354 23858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:38.852969 23858 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:38.853071 23858 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:38.853165 23858 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:38.853227 23858 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:38.886307 23858 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.133:37217
I20240503 08:11:38.886361 23984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.133:37217 every 8 connection(s)
I20240503 08:11:38.887600 23858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:38.897902 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 23858
I20240503 08:11:38.898078 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:11:38.900281 23985 heartbeater.cc:347] Connected to a master server at 127.20.107.190:34385
I20240503 08:11:38.900573 23985 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:38.901437 23985 heartbeater.cc:511] Master 127.20.107.190:34385 requested a full tablet report, sending...
I20240503 08:11:38.902776 23277 ts_manager.cc:194] Registered new tserver with Master: 44375967cba843c1ae87fa471159b6c2 (127.20.107.133:37217)
I20240503 08:11:38.903558 23277 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.133:38045
I20240503 08:11:38.914139 20910 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:11:38.929529 23277 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54582:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 08:11:38.962775 23790 tablet_service.cc:1467] Processing CreateTablet for tablet e00fbdaf60044377b152ede2ef20e01f (DEFAULT_TABLE table=TestTable [id=78ef580bc34c41c39156431bcb087ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:38.964246 23790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00fbdaf60044377b152ede2ef20e01f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:38.965457 23399 tablet_service.cc:1467] Processing CreateTablet for tablet e00fbdaf60044377b152ede2ef20e01f (DEFAULT_TABLE table=TestTable [id=78ef580bc34c41c39156431bcb087ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:38.966012 23530 tablet_service.cc:1467] Processing CreateTablet for tablet e00fbdaf60044377b152ede2ef20e01f (DEFAULT_TABLE table=TestTable [id=78ef580bc34c41c39156431bcb087ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:38.967085 23399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00fbdaf60044377b152ede2ef20e01f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:38.967582 23530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00fbdaf60044377b152ede2ef20e01f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:38.977826 24003 tablet_bootstrap.cc:492] T e00fbdaf60044377b152ede2ef20e01f P e517a0e8f9ac44daaa6e9be4cc47a2c1: Bootstrap starting.
I20240503 08:11:38.980841 24003 tablet_bootstrap.cc:654] T e00fbdaf60044377b152ede2ef20e01f P e517a0e8f9ac44daaa6e9be4cc47a2c1: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:38.981703 24004 tablet_bootstrap.cc:492] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e: Bootstrap starting.
I20240503 08:11:38.982887 24003 log.cc:824] T e00fbdaf60044377b152ede2ef20e01f P e517a0e8f9ac44daaa6e9be4cc47a2c1: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:38.985553 24004 tablet_bootstrap.cc:654] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:38.985632 23725 heartbeater.cc:503] Master 127.20.107.190:34385 was elected leader, sending a full tablet report...
I20240503 08:11:38.985893 24005 tablet_bootstrap.cc:492] T e00fbdaf60044377b152ede2ef20e01f P d8d4bc8fff4042b69580a73e50eb743e: Bootstrap starting.
I20240503 08:11:38.987039 24004 log.cc:824] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:38.989076 24005 tablet_bootstrap.cc:654] T e00fbdaf60044377b152ede2ef20e01f P d8d4bc8fff4042b69580a73e50eb743e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:38.989987 24005 log.cc:824] T e00fbdaf60044377b152ede2ef20e01f P d8d4bc8fff4042b69580a73e50eb743e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:38.992605 24005 tablet_bootstrap.cc:492] T e00fbdaf60044377b152ede2ef20e01f P d8d4bc8fff4042b69580a73e50eb743e: No bootstrap required, opened a new log
I20240503 08:11:38.992908 24005 ts_tablet_manager.cc:1367] T e00fbdaf60044377b152ede2ef20e01f P d8d4bc8fff4042b69580a73e50eb743e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 08:11:38.994042 24003 tablet_bootstrap.cc:492] T e00fbdaf60044377b152ede2ef20e01f P e517a0e8f9ac44daaa6e9be4cc47a2c1: No bootstrap required, opened a new log
I20240503 08:11:38.994289 24003 ts_tablet_manager.cc:1367] T e00fbdaf60044377b152ede2ef20e01f P e517a0e8f9ac44daaa6e9be4cc47a2c1: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.008s
I20240503 08:11:38.996651 24004 tablet_bootstrap.cc:492] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e: No bootstrap required, opened a new log
I20240503 08:11:38.996958 24004 ts_tablet_manager.cc:1367] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20240503 08:11:38.999811 24005 raft_consensus.cc:348] T e00fbdaf60044377b152ede2ef20e01f P d8d4bc8fff4042b69580a73e50eb743e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1141dfda05b543e484c0bc5065da7f3e" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 44625 } } peers { permanent_uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 38361 } } peers { permanent_uuid: "d8d4bc8fff4042b69580a73e50eb743e" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 34923 } }
I20240503 08:11:39.000150 24005 raft_consensus.cc:374] T e00fbdaf60044377b152ede2ef20e01f P d8d4bc8fff4042b69580a73e50eb743e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:39.000242 24005 raft_consensus.cc:729] T e00fbdaf60044377b152ede2ef20e01f P d8d4bc8fff4042b69580a73e50eb743e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d4bc8fff4042b69580a73e50eb743e, State: Initialized, Role: FOLLOWER
I20240503 08:11:39.000887 24005 consensus_queue.cc:260] T e00fbdaf60044377b152ede2ef20e01f P d8d4bc8fff4042b69580a73e50eb743e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1141dfda05b543e484c0bc5065da7f3e" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 44625 } } peers { permanent_uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 38361 } } peers { permanent_uuid: "d8d4bc8fff4042b69580a73e50eb743e" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 34923 } }
I20240503 08:11:39.002195 24003 raft_consensus.cc:348] T e00fbdaf60044377b152ede2ef20e01f P e517a0e8f9ac44daaa6e9be4cc47a2c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1141dfda05b543e484c0bc5065da7f3e" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 44625 } } peers { permanent_uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 38361 } } peers { permanent_uuid: "d8d4bc8fff4042b69580a73e50eb743e" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 34923 } }
I20240503 08:11:39.002621 24003 raft_consensus.cc:374] T e00fbdaf60044377b152ede2ef20e01f P e517a0e8f9ac44daaa6e9be4cc47a2c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:39.002724 24003 raft_consensus.cc:729] T e00fbdaf60044377b152ede2ef20e01f P e517a0e8f9ac44daaa6e9be4cc47a2c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517a0e8f9ac44daaa6e9be4cc47a2c1, State: Initialized, Role: FOLLOWER
I20240503 08:11:39.003374 24003 consensus_queue.cc:260] T e00fbdaf60044377b152ede2ef20e01f P e517a0e8f9ac44daaa6e9be4cc47a2c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1141dfda05b543e484c0bc5065da7f3e" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 44625 } } peers { permanent_uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 38361 } } peers { permanent_uuid: "d8d4bc8fff4042b69580a73e50eb743e" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 34923 } }
I20240503 08:11:39.006234 24005 ts_tablet_manager.cc:1398] T e00fbdaf60044377b152ede2ef20e01f P d8d4bc8fff4042b69580a73e50eb743e: Time spent starting tablet: real 0.013s user 0.002s sys 0.010s
I20240503 08:11:39.006544 24004 raft_consensus.cc:348] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1141dfda05b543e484c0bc5065da7f3e" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 44625 } } peers { permanent_uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 38361 } } peers { permanent_uuid: "d8d4bc8fff4042b69580a73e50eb743e" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 34923 } }
I20240503 08:11:39.006943 24004 raft_consensus.cc:374] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:39.006992 23855 heartbeater.cc:503] Master 127.20.107.190:34385 was elected leader, sending a full tablet report...
I20240503 08:11:39.007056 24004 raft_consensus.cc:729] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1141dfda05b543e484c0bc5065da7f3e, State: Initialized, Role: FOLLOWER
I20240503 08:11:39.007684 24004 consensus_queue.cc:260] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1141dfda05b543e484c0bc5065da7f3e" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 44625 } } peers { permanent_uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 38361 } } peers { permanent_uuid: "d8d4bc8fff4042b69580a73e50eb743e" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 34923 } }
I20240503 08:11:39.009464 24003 ts_tablet_manager.cc:1398] T e00fbdaf60044377b152ede2ef20e01f P e517a0e8f9ac44daaa6e9be4cc47a2c1: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
W20240503 08:11:39.011831 23596 tablet.cc:2369] T e00fbdaf60044377b152ede2ef20e01f P d8d4bc8fff4042b69580a73e50eb743e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:39.011879 24004 ts_tablet_manager.cc:1398] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20240503 08:11:39.047396 23465 tablet.cc:2369] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:39.172327 23856 tablet.cc:2369] T e00fbdaf60044377b152ede2ef20e01f P e517a0e8f9ac44daaa6e9be4cc47a2c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:39.274668 24011 raft_consensus.cc:482] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:39.274972 24011 raft_consensus.cc:504] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1141dfda05b543e484c0bc5065da7f3e" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 44625 } } peers { permanent_uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 38361 } } peers { permanent_uuid: "d8d4bc8fff4042b69580a73e50eb743e" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 34923 } }
I20240503 08:11:39.276718 24011 leader_election.cc:288] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e517a0e8f9ac44daaa6e9be4cc47a2c1 (127.20.107.132:38361), d8d4bc8fff4042b69580a73e50eb743e (127.20.107.130:34923)
I20240503 08:11:39.283793 23810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00fbdaf60044377b152ede2ef20e01f" candidate_uuid: "1141dfda05b543e484c0bc5065da7f3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1" is_pre_election: true
I20240503 08:11:39.283794 23550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00fbdaf60044377b152ede2ef20e01f" candidate_uuid: "1141dfda05b543e484c0bc5065da7f3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d4bc8fff4042b69580a73e50eb743e" is_pre_election: true
I20240503 08:11:39.284245 23810 raft_consensus.cc:2454] T e00fbdaf60044377b152ede2ef20e01f P e517a0e8f9ac44daaa6e9be4cc47a2c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1141dfda05b543e484c0bc5065da7f3e in term 0.
I20240503 08:11:39.284255 23550 raft_consensus.cc:2454] T e00fbdaf60044377b152ede2ef20e01f P d8d4bc8fff4042b69580a73e50eb743e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1141dfda05b543e484c0bc5065da7f3e in term 0.
I20240503 08:11:39.284840 23353 leader_election.cc:302] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1141dfda05b543e484c0bc5065da7f3e, e517a0e8f9ac44daaa6e9be4cc47a2c1; no voters:
I20240503 08:11:39.285260 24011 raft_consensus.cc:2781] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:39.285394 24011 raft_consensus.cc:482] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:39.285494 24011 raft_consensus.cc:3037] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:39.287524 24011 raft_consensus.cc:504] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1141dfda05b543e484c0bc5065da7f3e" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 44625 } } peers { permanent_uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 38361 } } peers { permanent_uuid: "d8d4bc8fff4042b69580a73e50eb743e" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 34923 } }
I20240503 08:11:39.288093 24011 leader_election.cc:288] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [CANDIDATE]: Term 1 election: Requested vote from peers e517a0e8f9ac44daaa6e9be4cc47a2c1 (127.20.107.132:38361), d8d4bc8fff4042b69580a73e50eb743e (127.20.107.130:34923)
I20240503 08:11:39.288550 23810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00fbdaf60044377b152ede2ef20e01f" candidate_uuid: "1141dfda05b543e484c0bc5065da7f3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1"
I20240503 08:11:39.288611 23550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00fbdaf60044377b152ede2ef20e01f" candidate_uuid: "1141dfda05b543e484c0bc5065da7f3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d4bc8fff4042b69580a73e50eb743e"
I20240503 08:11:39.288738 23810 raft_consensus.cc:3037] T e00fbdaf60044377b152ede2ef20e01f P e517a0e8f9ac44daaa6e9be4cc47a2c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:39.288810 23550 raft_consensus.cc:3037] T e00fbdaf60044377b152ede2ef20e01f P d8d4bc8fff4042b69580a73e50eb743e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:39.290813 23810 raft_consensus.cc:2454] T e00fbdaf60044377b152ede2ef20e01f P e517a0e8f9ac44daaa6e9be4cc47a2c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1141dfda05b543e484c0bc5065da7f3e in term 1.
I20240503 08:11:39.290871 23550 raft_consensus.cc:2454] T e00fbdaf60044377b152ede2ef20e01f P d8d4bc8fff4042b69580a73e50eb743e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1141dfda05b543e484c0bc5065da7f3e in term 1.
I20240503 08:11:39.291194 23353 leader_election.cc:302] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1141dfda05b543e484c0bc5065da7f3e, e517a0e8f9ac44daaa6e9be4cc47a2c1; no voters:
I20240503 08:11:39.291507 24011 raft_consensus.cc:2781] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:39.291950 24011 raft_consensus.cc:686] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [term 1 LEADER]: Becoming Leader. State: Replica: 1141dfda05b543e484c0bc5065da7f3e, State: Running, Role: LEADER
I20240503 08:11:39.292454 24011 consensus_queue.cc:237] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1141dfda05b543e484c0bc5065da7f3e" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 44625 } } peers { permanent_uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 38361 } } peers { permanent_uuid: "d8d4bc8fff4042b69580a73e50eb743e" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 34923 } }
I20240503 08:11:39.296833 23278 catalog_manager.cc:5484] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1141dfda05b543e484c0bc5065da7f3e (127.20.107.129). New cstate: current_term: 1 leader_uuid: "1141dfda05b543e484c0bc5065da7f3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1141dfda05b543e484c0bc5065da7f3e" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 44625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 38361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d4bc8fff4042b69580a73e50eb743e" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 34923 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:39.323616 20910 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:11:39.325758 20910 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1141dfda05b543e484c0bc5065da7f3e to finish bootstrapping
I20240503 08:11:39.334389 20910 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d8d4bc8fff4042b69580a73e50eb743e to finish bootstrapping
I20240503 08:11:39.342288 20910 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f05cb769503641fcaf7fc8c2e9250b1a to finish bootstrapping
I20240503 08:11:39.351095 20910 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e517a0e8f9ac44daaa6e9be4cc47a2c1 to finish bootstrapping
I20240503 08:11:39.358634 20910 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 44375967cba843c1ae87fa471159b6c2 to finish bootstrapping
I20240503 08:11:39.759269 24011 consensus_queue.cc:1035] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d4bc8fff4042b69580a73e50eb743e" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 34923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:39.772132 24014 consensus_queue.cc:1035] T e00fbdaf60044377b152ede2ef20e01f P 1141dfda05b543e484c0bc5065da7f3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517a0e8f9ac44daaa6e9be4cc47a2c1" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 38361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:39.867813 23399 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:39.869633 23920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:39.871865 23790 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:39.873134 23530 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:39.876924 23660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:39.905911 23985 heartbeater.cc:503] Master 127.20.107.190:34385 was elected leader, sending a full tablet report...
I20240503 08:11:40.049716 23399 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:40.060041 23920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.070719 23790 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.071100 23530 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.078770 23660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.705762 23530 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.705952 23399 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:40.713001 23920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.713203 23790 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.722833 23660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.884969 23399 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:40.888710 23920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.894289 23530 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.895895 23790 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.912228 23660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:41.088577 23399 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:41.099198 23920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:41.100282 23790 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:41.106884 23530 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:41.114456 23660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:41.264940 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 23337
I20240503 08:11:41.266237 23458 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:41.360767 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 23337
W20240503 08:11:41.380236 23484 connection.cc:535] server connection from 127.20.107.129:36061 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:11:41.380558 23265 connection.cc:535] server connection from 127.20.107.129:44631 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:11:41.380595 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 23468
I20240503 08:11:41.381812 23589 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:41.484550 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 23468
I20240503 08:11:41.502487 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 23598
I20240503 08:11:41.503551 23719 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:41.594144 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 23598
I20240503 08:11:41.612016 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 23728
I20240503 08:11:41.613095 23850 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:41.708334 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 23728
I20240503 08:11:41.726167 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 23858
I20240503 08:11:41.727131 23979 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:41.817044 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 23858
I20240503 08:11:41.835937 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 23247
I20240503 08:11:41.836973 23308 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:41.950230 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 23247
2024-05-03T08:11:41Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (5982 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5982 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/3
I20240503 08:11:41.994303 20910 test_util.cc:268] Using random seed: 1158739236
I20240503 08:11:41.996114 20910 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:41.996176 20910 ts_itest-base.cc:113] --------------
I20240503 08:11:41.996232 20910 ts_itest-base.cc:114] 7 tablet servers
I20240503 08:11:41.996282 20910 ts_itest-base.cc:115] 2 replicas per TS
I20240503 08:11:41.996351 20910 ts_itest-base.cc:116] --------------
2024-05-03T08:11:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:42Z Disabled control of system clock
I20240503 08:11:42.029907 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.107.190:44243
--webserver_interface=127.20.107.190
--webserver_port=0
--builtin_ntp_servers=127.20.107.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.107.190:44243
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 08:11:42.353713 24214 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 08:11:42.354076 24214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:42.354194 24214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:42.354310 24214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:42.361783 24214 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:42.361920 24214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:11:42.362028 24214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:42.362099 24214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:11:42.362192 24214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:11:42.371716 24214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.107.190:44243
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.107.190:44243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.107.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:42.373503 24214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:42.375505 24214 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:42.387089 24219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:42.387771 24220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:42.388659 24222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:42.389585 24214 server_base.cc:1034] running on GCE node
I20240503 08:11:42.390452 24214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:42.391945 24214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:42.393162 24214 hybrid_clock.cc:648] HybridClock initialized: now 1714723902393108 us; error 37 us; skew 500 ppm
I20240503 08:11:42.393487 24214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:42.399782 24214 webserver.cc:460] Webserver started at http://127.20.107.190:35741/ using document root <none> and password file <none>
I20240503 08:11:42.400534 24214 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:42.400677 24214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:42.401028 24214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:42.403659 24214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4a981765e30a453eab8294095153bf23"
format_stamp: "Formatted at 2024-05-03 08:11:42 on dist-test-slave-4tl1"
I20240503 08:11:42.404536 24214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4a981765e30a453eab8294095153bf23"
format_stamp: "Formatted at 2024-05-03 08:11:42 on dist-test-slave-4tl1"
I20240503 08:11:42.410215 24214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20240503 08:11:42.413559 24228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:42.414976 24214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:42.415166 24214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4a981765e30a453eab8294095153bf23"
format_stamp: "Formatted at 2024-05-03 08:11:42 on dist-test-slave-4tl1"
I20240503 08:11:42.415359 24214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:42.447584 24214 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:42.448480 24214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:42.448776 24214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:42.467353 24214 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.190:44243
I20240503 08:11:42.467389 24279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.190:44243 every 8 connection(s)
I20240503 08:11:42.468775 24214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:11:42.472211 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 24214
I20240503 08:11:42.472453 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:11:42.472843 24280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:42.485035 24280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23: Bootstrap starting.
I20240503 08:11:42.488113 24280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:42.489317 24280 log.cc:824] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:42.491888 24280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23: No bootstrap required, opened a new log
I20240503 08:11:42.498391 24280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a981765e30a453eab8294095153bf23" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 44243 } }
I20240503 08:11:42.498751 24280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:42.498850 24280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a981765e30a453eab8294095153bf23, State: Initialized, Role: FOLLOWER
I20240503 08:11:42.499495 24280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a981765e30a453eab8294095153bf23" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 44243 } }
I20240503 08:11:42.499697 24280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:42.499816 24280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:42.499961 24280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:42.502352 24280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a981765e30a453eab8294095153bf23" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 44243 } }
I20240503 08:11:42.502835 24280 leader_election.cc:302] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a981765e30a453eab8294095153bf23; no voters:
I20240503 08:11:42.503433 24280 leader_election.cc:288] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:42.503571 24285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:42.504194 24285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [term 1 LEADER]: Becoming Leader. State: Replica: 4a981765e30a453eab8294095153bf23, State: Running, Role: LEADER
I20240503 08:11:42.504750 24285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a981765e30a453eab8294095153bf23" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 44243 } }
I20240503 08:11:42.505429 24280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:42.507751 24286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a981765e30a453eab8294095153bf23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a981765e30a453eab8294095153bf23" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 44243 } } }
I20240503 08:11:42.508051 24286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:42.508333 24287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a981765e30a453eab8294095153bf23. Latest consensus state: current_term: 1 leader_uuid: "4a981765e30a453eab8294095153bf23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a981765e30a453eab8294095153bf23" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 44243 } } }
I20240503 08:11:42.508595 24287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:42.510615 24290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:42.515327 24290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:42.524336 24290 catalog_manager.cc:1347] Generated new cluster ID: 155184c701b8439dbc7d4fb9338edcd2
I20240503 08:11:42.524464 24290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:42.537047 24290 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:42.538789 24290 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:42.548480 24290 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4a981765e30a453eab8294095153bf23: Generated new TSK 0
I20240503 08:11:42.549569 24290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:42.563670 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.129:0
--local_ip_for_outbound_sockets=127.20.107.129
--webserver_interface=127.20.107.129
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--builtin_ntp_servers=127.20.107.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:11:42.897246 24304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:42.897574 24304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:42.897667 24304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:42.905601 24304 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:42.905742 24304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:42.905870 24304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.129
I20240503 08:11:42.916668 24304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.107.129
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.129
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:42.918077 24304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:42.920043 24304 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:42.933275 24309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:42.933871 24310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:42.934634 24312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:42.935410 24304 server_base.cc:1034] running on GCE node
I20240503 08:11:42.936038 24304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:42.937055 24304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:42.938271 24304 hybrid_clock.cc:648] HybridClock initialized: now 1714723902938223 us; error 43 us; skew 500 ppm
I20240503 08:11:42.938601 24304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:42.940932 24304 webserver.cc:460] Webserver started at http://127.20.107.129:36015/ using document root <none> and password file <none>
I20240503 08:11:42.941694 24304 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:42.941814 24304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:42.942127 24304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:42.944830 24304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b53502e2a1ae4c86a054d38048dd0df6"
format_stamp: "Formatted at 2024-05-03 08:11:42 on dist-test-slave-4tl1"
I20240503 08:11:42.945637 24304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b53502e2a1ae4c86a054d38048dd0df6"
format_stamp: "Formatted at 2024-05-03 08:11:42 on dist-test-slave-4tl1"
I20240503 08:11:42.950706 24304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:11:42.954339 24318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:42.955653 24304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 08:11:42.955812 24304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b53502e2a1ae4c86a054d38048dd0df6"
format_stamp: "Formatted at 2024-05-03 08:11:42 on dist-test-slave-4tl1"
I20240503 08:11:42.956010 24304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:42.983038 24304 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:42.983923 24304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:42.984200 24304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:42.985266 24304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:42.986766 24304 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:42.986864 24304 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:42.986979 24304 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:42.987052 24304 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:43.018282 24304 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.129:46775
I20240503 08:11:43.018328 24430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.129:46775 every 8 connection(s)
I20240503 08:11:43.019624 24304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:43.027295 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 24304
I20240503 08:11:43.027508 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:11:43.030783 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.130:0
--local_ip_for_outbound_sockets=127.20.107.130
--webserver_interface=127.20.107.130
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--builtin_ntp_servers=127.20.107.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:43.032555 24431 heartbeater.cc:347] Connected to a master server at 127.20.107.190:44243
I20240503 08:11:43.032860 24431 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:43.033726 24431 heartbeater.cc:511] Master 127.20.107.190:44243 requested a full tablet report, sending...
I20240503 08:11:43.036087 24245 ts_manager.cc:194] Registered new tserver with Master: b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:43.038156 24245 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.129:40257
W20240503 08:11:43.365906 24435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:43.366226 24435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:43.366339 24435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:43.374377 24435 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:43.374531 24435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:43.374699 24435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.130
I20240503 08:11:43.383946 24435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.107.130
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.130
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:43.385295 24435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:43.387009 24435 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:43.399662 24440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:43.400486 24441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:43.401636 24443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:43.402480 24435 server_base.cc:1034] running on GCE node
I20240503 08:11:43.403069 24435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:43.404044 24435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:43.405242 24435 hybrid_clock.cc:648] HybridClock initialized: now 1714723903405196 us; error 34 us; skew 500 ppm
I20240503 08:11:43.405580 24435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:43.407827 24435 webserver.cc:460] Webserver started at http://127.20.107.130:37441/ using document root <none> and password file <none>
I20240503 08:11:43.408679 24435 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:43.408820 24435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:43.409180 24435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:43.411849 24435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6a640901590545eeb69e216d2399ea51"
format_stamp: "Formatted at 2024-05-03 08:11:43 on dist-test-slave-4tl1"
I20240503 08:11:43.412595 24435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6a640901590545eeb69e216d2399ea51"
format_stamp: "Formatted at 2024-05-03 08:11:43 on dist-test-slave-4tl1"
I20240503 08:11:43.417892 24435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240503 08:11:43.421922 24449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:43.423341 24435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240503 08:11:43.423497 24435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6a640901590545eeb69e216d2399ea51"
format_stamp: "Formatted at 2024-05-03 08:11:43 on dist-test-slave-4tl1"
I20240503 08:11:43.423702 24435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:43.447726 24435 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:43.448630 24435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:43.448959 24435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:43.449929 24435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:43.451434 24435 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:43.451535 24435 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:43.451650 24435 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:43.451721 24435 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:43.484270 24435 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.130:43793
I20240503 08:11:43.484283 24561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.130:43793 every 8 connection(s)
I20240503 08:11:43.486004 24435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:11:43.495882 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 24435
I20240503 08:11:43.496076 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:11:43.499635 24562 heartbeater.cc:347] Connected to a master server at 127.20.107.190:44243
I20240503 08:11:43.499650 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.131:0
--local_ip_for_outbound_sockets=127.20.107.131
--webserver_interface=127.20.107.131
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--builtin_ntp_servers=127.20.107.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:43.499992 24562 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:43.500834 24562 heartbeater.cc:511] Master 127.20.107.190:44243 requested a full tablet report, sending...
I20240503 08:11:43.502429 24245 ts_manager.cc:194] Registered new tserver with Master: 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:11:43.503558 24245 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.130:53587
W20240503 08:11:43.837270 24565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:43.837565 24565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:43.837661 24565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:43.845609 24565 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:43.845746 24565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:43.845868 24565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.131
I20240503 08:11:43.854984 24565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.107.131
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.131
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:43.856266 24565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:43.858000 24565 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:43.870738 24571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:43.871567 24573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:43.872442 24570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:43.873418 24565 server_base.cc:1034] running on GCE node
I20240503 08:11:43.874053 24565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:43.875043 24565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:43.876264 24565 hybrid_clock.cc:648] HybridClock initialized: now 1714723903876213 us; error 38 us; skew 500 ppm
I20240503 08:11:43.876627 24565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:43.884603 24565 webserver.cc:460] Webserver started at http://127.20.107.131:41549/ using document root <none> and password file <none>
I20240503 08:11:43.885434 24565 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:43.885555 24565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:43.885879 24565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:43.888466 24565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "56fb8e40a4da4b3794f07ad8137ecf42"
format_stamp: "Formatted at 2024-05-03 08:11:43 on dist-test-slave-4tl1"
I20240503 08:11:43.889394 24565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "56fb8e40a4da4b3794f07ad8137ecf42"
format_stamp: "Formatted at 2024-05-03 08:11:43 on dist-test-slave-4tl1"
I20240503 08:11:43.894532 24565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 08:11:43.897764 24579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:43.899101 24565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:43.899255 24565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "56fb8e40a4da4b3794f07ad8137ecf42"
format_stamp: "Formatted at 2024-05-03 08:11:43 on dist-test-slave-4tl1"
I20240503 08:11:43.899449 24565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:43.917739 24565 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:43.918571 24565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:43.918855 24565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:43.919965 24565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:43.921625 24565 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:43.921727 24565 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:43.921844 24565 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:43.921913 24565 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:43.953687 24565 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.131:34003
I20240503 08:11:43.953701 24691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.131:34003 every 8 connection(s)
I20240503 08:11:43.955103 24565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:43.964617 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 24565
I20240503 08:11:43.964839 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:11:43.971992 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.132:0
--local_ip_for_outbound_sockets=127.20.107.132
--webserver_interface=127.20.107.132
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--builtin_ntp_servers=127.20.107.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:43.972299 24692 heartbeater.cc:347] Connected to a master server at 127.20.107.190:44243
I20240503 08:11:43.972699 24692 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:43.973688 24692 heartbeater.cc:511] Master 127.20.107.190:44243 requested a full tablet report, sending...
I20240503 08:11:43.975320 24245 ts_manager.cc:194] Registered new tserver with Master: 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:11:43.976102 24245 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.131:59511
I20240503 08:11:44.041033 24431 heartbeater.cc:503] Master 127.20.107.190:44243 was elected leader, sending a full tablet report...
W20240503 08:11:44.326608 24695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:44.326917 24695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:44.327013 24695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:44.334738 24695 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:44.334869 24695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:44.335042 24695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.132
I20240503 08:11:44.344061 24695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.107.132
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.132
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:44.345436 24695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:44.347142 24695 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:44.359768 24700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:44.360651 24703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:44.361420 24701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:44.362394 24695 server_base.cc:1034] running on GCE node
I20240503 08:11:44.363013 24695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:44.363988 24695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:44.365213 24695 hybrid_clock.cc:648] HybridClock initialized: now 1714723904365161 us; error 40 us; skew 500 ppm
I20240503 08:11:44.365585 24695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:44.368194 24695 webserver.cc:460] Webserver started at http://127.20.107.132:33449/ using document root <none> and password file <none>
I20240503 08:11:44.369012 24695 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:44.369135 24695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:44.369458 24695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:44.372200 24695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e7416e7c03a74a24b8fe2508111770d7"
format_stamp: "Formatted at 2024-05-03 08:11:44 on dist-test-slave-4tl1"
I20240503 08:11:44.373034 24695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e7416e7c03a74a24b8fe2508111770d7"
format_stamp: "Formatted at 2024-05-03 08:11:44 on dist-test-slave-4tl1"
I20240503 08:11:44.378407 24695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 08:11:44.381884 24709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:44.383239 24695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:44.383401 24695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e7416e7c03a74a24b8fe2508111770d7"
format_stamp: "Formatted at 2024-05-03 08:11:44 on dist-test-slave-4tl1"
I20240503 08:11:44.383598 24695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:44.399379 24695 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:44.400206 24695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:44.400473 24695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:44.401458 24695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:44.402942 24695 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:44.403039 24695 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:44.403153 24695 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:44.403223 24695 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:44.434388 24695 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.132:36475
I20240503 08:11:44.434437 24821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.132:36475 every 8 connection(s)
I20240503 08:11:44.435663 24695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:44.437228 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 24695
I20240503 08:11:44.437379 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:11:44.440582 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.133:0
--local_ip_for_outbound_sockets=127.20.107.133
--webserver_interface=127.20.107.133
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--builtin_ntp_servers=127.20.107.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:44.448606 24822 heartbeater.cc:347] Connected to a master server at 127.20.107.190:44243
I20240503 08:11:44.448925 24822 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:44.449726 24822 heartbeater.cc:511] Master 127.20.107.190:44243 requested a full tablet report, sending...
I20240503 08:11:44.451037 24245 ts_manager.cc:194] Registered new tserver with Master: e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132:36475)
I20240503 08:11:44.451803 24245 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.132:57703
I20240503 08:11:44.506626 24562 heartbeater.cc:503] Master 127.20.107.190:44243 was elected leader, sending a full tablet report...
W20240503 08:11:44.776777 24825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:44.777140 24825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:44.777264 24825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:44.784734 24825 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:44.784922 24825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:44.785094 24825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.133
I20240503 08:11:44.794399 24825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.107.133
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.133
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:44.795845 24825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:44.797642 24825 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:44.810230 24830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:44.810832 24831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:44.811688 24833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:44.812562 24825 server_base.cc:1034] running on GCE node
I20240503 08:11:44.813251 24825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:44.814169 24825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:44.815361 24825 hybrid_clock.cc:648] HybridClock initialized: now 1714723904815314 us; error 48 us; skew 500 ppm
I20240503 08:11:44.815676 24825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:44.817916 24825 webserver.cc:460] Webserver started at http://127.20.107.133:45567/ using document root <none> and password file <none>
I20240503 08:11:44.818609 24825 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:44.818707 24825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:44.818985 24825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:44.821681 24825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f9705fdde3854639ba85097a161f37e7"
format_stamp: "Formatted at 2024-05-03 08:11:44 on dist-test-slave-4tl1"
I20240503 08:11:44.822386 24825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f9705fdde3854639ba85097a161f37e7"
format_stamp: "Formatted at 2024-05-03 08:11:44 on dist-test-slave-4tl1"
I20240503 08:11:44.827543 24825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:11:44.830938 24839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:44.832245 24825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:44.832402 24825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f9705fdde3854639ba85097a161f37e7"
format_stamp: "Formatted at 2024-05-03 08:11:44 on dist-test-slave-4tl1"
I20240503 08:11:44.832592 24825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:44.858295 24825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:44.859202 24825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:44.859483 24825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:44.860550 24825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:44.862181 24825 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:44.862282 24825 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:44.862397 24825 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:44.862470 24825 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:44.893765 24825 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.133:45569
I20240503 08:11:44.893823 24951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.133:45569 every 8 connection(s)
I20240503 08:11:44.895222 24825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:44.904629 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 24825
I20240503 08:11:44.904860 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:11:44.907720 24952 heartbeater.cc:347] Connected to a master server at 127.20.107.190:44243
I20240503 08:11:44.908090 24952 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:44.908372 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.134:0
--local_ip_for_outbound_sockets=127.20.107.134
--webserver_interface=127.20.107.134
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--builtin_ntp_servers=127.20.107.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:44.909188 24952 heartbeater.cc:511] Master 127.20.107.190:44243 requested a full tablet report, sending...
I20240503 08:11:44.910727 24245 ts_manager.cc:194] Registered new tserver with Master: f9705fdde3854639ba85097a161f37e7 (127.20.107.133:45569)
I20240503 08:11:44.911494 24245 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.133:60435
I20240503 08:11:44.978897 24692 heartbeater.cc:503] Master 127.20.107.190:44243 was elected leader, sending a full tablet report...
W20240503 08:11:45.248737 24955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:45.249109 24955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:45.249259 24955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:45.256698 24955 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:45.256857 24955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:45.257061 24955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.134
I20240503 08:11:45.267390 24955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.107.134
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.134
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:45.269016 24955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:45.270782 24955 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:45.285478 24963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:45.286190 24960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:45.287083 24961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:45.287961 24955 server_base.cc:1034] running on GCE node
I20240503 08:11:45.288683 24955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:45.289716 24955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:45.290935 24955 hybrid_clock.cc:648] HybridClock initialized: now 1714723905290885 us; error 36 us; skew 500 ppm
I20240503 08:11:45.291307 24955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:45.293602 24955 webserver.cc:460] Webserver started at http://127.20.107.134:37899/ using document root <none> and password file <none>
I20240503 08:11:45.294343 24955 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:45.294446 24955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:45.294775 24955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:45.297374 24955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9d313001b0a647caba8f53540a2b67ac"
format_stamp: "Formatted at 2024-05-03 08:11:45 on dist-test-slave-4tl1"
I20240503 08:11:45.298120 24955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9d313001b0a647caba8f53540a2b67ac"
format_stamp: "Formatted at 2024-05-03 08:11:45 on dist-test-slave-4tl1"
I20240503 08:11:45.303087 24955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240503 08:11:45.306524 24969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:45.307883 24955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 08:11:45.308068 24955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9d313001b0a647caba8f53540a2b67ac"
format_stamp: "Formatted at 2024-05-03 08:11:45 on dist-test-slave-4tl1"
I20240503 08:11:45.308285 24955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:45.336828 24955 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:45.337836 24955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:45.338146 24955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:45.339159 24955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:45.340737 24955 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:45.340840 24955 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:45.340989 24955 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:45.341084 24955 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:45.375264 24955 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.134:46393
I20240503 08:11:45.375327 25081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.134:46393 every 8 connection(s)
I20240503 08:11:45.376619 24955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:11:45.384366 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 24955
I20240503 08:11:45.384531 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:11:45.388238 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.135:0
--local_ip_for_outbound_sockets=127.20.107.135
--webserver_interface=127.20.107.135
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--builtin_ntp_servers=127.20.107.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:45.389328 25082 heartbeater.cc:347] Connected to a master server at 127.20.107.190:44243
I20240503 08:11:45.389689 25082 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:45.390726 25082 heartbeater.cc:511] Master 127.20.107.190:44243 requested a full tablet report, sending...
I20240503 08:11:45.392264 24245 ts_manager.cc:194] Registered new tserver with Master: 9d313001b0a647caba8f53540a2b67ac (127.20.107.134:46393)
I20240503 08:11:45.393057 24245 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.134:46165
I20240503 08:11:45.454447 24822 heartbeater.cc:503] Master 127.20.107.190:44243 was elected leader, sending a full tablet report...
W20240503 08:11:45.725500 25085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:45.725838 25085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:45.725982 25085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:45.733698 25085 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:45.733847 25085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:45.734010 25085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.135
I20240503 08:11:45.743131 25085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.107.135
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.135
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:45.744446 25085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:45.746183 25085 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:45.758724 25090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:45.759411 25091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:45.760383 25093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:45.761289 25085 server_base.cc:1034] running on GCE node
I20240503 08:11:45.761960 25085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:45.762941 25085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:45.764163 25085 hybrid_clock.cc:648] HybridClock initialized: now 1714723905764126 us; error 36 us; skew 500 ppm
I20240503 08:11:45.764495 25085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:45.766921 25085 webserver.cc:460] Webserver started at http://127.20.107.135:36541/ using document root <none> and password file <none>
I20240503 08:11:45.767659 25085 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:45.767786 25085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:45.768105 25085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:45.770807 25085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "a0d221e07029452cae34b8920884507e"
format_stamp: "Formatted at 2024-05-03 08:11:45 on dist-test-slave-4tl1"
I20240503 08:11:45.771598 25085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "a0d221e07029452cae34b8920884507e"
format_stamp: "Formatted at 2024-05-03 08:11:45 on dist-test-slave-4tl1"
I20240503 08:11:45.776932 25085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240503 08:11:45.780746 25099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:45.782366 25085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240503 08:11:45.782570 25085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a0d221e07029452cae34b8920884507e"
format_stamp: "Formatted at 2024-05-03 08:11:45 on dist-test-slave-4tl1"
I20240503 08:11:45.782809 25085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:45.806759 25085 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:45.807986 25085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:45.808323 25085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:45.809638 25085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:45.811635 25085 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:45.811734 25085 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:45.811856 25085 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:45.811941 25085 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:45.843621 25085 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.135:45329
I20240503 08:11:45.843679 25211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.135:45329 every 8 connection(s)
I20240503 08:11:45.844944 25085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:11:45.853449 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 25085
I20240503 08:11:45.853688 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:11:45.857677 25212 heartbeater.cc:347] Connected to a master server at 127.20.107.190:44243
I20240503 08:11:45.857964 25212 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:45.858748 25212 heartbeater.cc:511] Master 127.20.107.190:44243 requested a full tablet report, sending...
I20240503 08:11:45.860270 24244 ts_manager.cc:194] Registered new tserver with Master: a0d221e07029452cae34b8920884507e (127.20.107.135:45329)
I20240503 08:11:45.861270 24244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.135:56333
I20240503 08:11:45.870440 20910 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 08:11:45.887425 24244 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58162:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 08:11:45.914568 24952 heartbeater.cc:503] Master 127.20.107.190:44243 was elected leader, sending a full tablet report...
I20240503 08:11:45.917166 25147 tablet_service.cc:1467] Processing CreateTablet for tablet 6d422797de9843edbc473fca1d8e9e87 (DEFAULT_TABLE table=TestTable [id=072979b49b8b4071923d58835e04c91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:45.917855 24497 tablet_service.cc:1467] Processing CreateTablet for tablet 6d422797de9843edbc473fca1d8e9e87 (DEFAULT_TABLE table=TestTable [id=072979b49b8b4071923d58835e04c91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:45.918900 25147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d422797de9843edbc473fca1d8e9e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:45.919291 24497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d422797de9843edbc473fca1d8e9e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:45.933528 25228 tablet_bootstrap.cc:492] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: Bootstrap starting.
I20240503 08:11:45.934458 25229 tablet_bootstrap.cc:492] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51: Bootstrap starting.
I20240503 08:11:45.937258 25228 tablet_bootstrap.cc:654] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:45.937849 25229 tablet_bootstrap.cc:654] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:45.938563 25228 log.cc:824] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:45.939043 25229 log.cc:824] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:45.941062 25228 tablet_bootstrap.cc:492] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: No bootstrap required, opened a new log
I20240503 08:11:45.941303 25228 ts_tablet_manager.cc:1367] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240503 08:11:45.941375 25229 tablet_bootstrap.cc:492] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51: No bootstrap required, opened a new log
I20240503 08:11:45.941614 25229 ts_tablet_manager.cc:1367] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 08:11:45.947486 25228 raft_consensus.cc:348] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:45.947494 25229 raft_consensus.cc:348] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:45.947809 25229 raft_consensus.cc:374] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:45.947822 25228 raft_consensus.cc:374] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:45.947921 25228 raft_consensus.cc:729] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d221e07029452cae34b8920884507e, State: Initialized, Role: FOLLOWER
I20240503 08:11:45.947913 25229 raft_consensus.cc:729] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Initialized, Role: FOLLOWER
I20240503 08:11:45.948496 25228 consensus_queue.cc:260] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:45.948491 25229 consensus_queue.cc:260] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:45.949599 25212 heartbeater.cc:503] Master 127.20.107.190:44243 was elected leader, sending a full tablet report...
I20240503 08:11:45.950223 25229 ts_tablet_manager.cc:1398] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20240503 08:11:45.951211 25228 ts_tablet_manager.cc:1398] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
W20240503 08:11:45.989367 24563 tablet.cc:2369] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:46.033267 25232 raft_consensus.cc:482] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:46.033542 25232 raft_consensus.cc:504] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:46.035043 25232 leader_election.cc:288] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:11:46.040827 24517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d422797de9843edbc473fca1d8e9e87" candidate_uuid: "a0d221e07029452cae34b8920884507e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a640901590545eeb69e216d2399ea51" is_pre_election: true
I20240503 08:11:46.041289 24517 raft_consensus.cc:2454] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d221e07029452cae34b8920884507e in term 0.
I20240503 08:11:46.041868 25101 leader_election.cc:302] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, a0d221e07029452cae34b8920884507e; no voters:
I20240503 08:11:46.042204 25232 raft_consensus.cc:2781] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:46.042351 25232 raft_consensus.cc:482] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:46.042448 25232 raft_consensus.cc:3037] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:46.044394 25232 raft_consensus.cc:504] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:46.044857 25232 leader_election.cc:288] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [CANDIDATE]: Term 1 election: Requested vote from peers 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:11:46.045356 24517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d422797de9843edbc473fca1d8e9e87" candidate_uuid: "a0d221e07029452cae34b8920884507e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a640901590545eeb69e216d2399ea51"
I20240503 08:11:46.045512 24517 raft_consensus.cc:3037] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:46.047761 24517 raft_consensus.cc:2454] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d221e07029452cae34b8920884507e in term 1.
I20240503 08:11:46.048103 25101 leader_election.cc:302] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, a0d221e07029452cae34b8920884507e; no voters:
I20240503 08:11:46.048372 25232 raft_consensus.cc:2781] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:46.048780 25232 raft_consensus.cc:686] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 1 LEADER]: Becoming Leader. State: Replica: a0d221e07029452cae34b8920884507e, State: Running, Role: LEADER
I20240503 08:11:46.049274 25232 consensus_queue.cc:237] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:46.053901 24244 catalog_manager.cc:5484] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d221e07029452cae34b8920884507e (127.20.107.135). New cstate: current_term: 1 leader_uuid: "a0d221e07029452cae34b8920884507e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:46.065177 20910 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 08:11:46.067612 20910 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b53502e2a1ae4c86a054d38048dd0df6 to finish bootstrapping
I20240503 08:11:46.078219 20910 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6a640901590545eeb69e216d2399ea51 to finish bootstrapping
I20240503 08:11:46.086561 20910 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 56fb8e40a4da4b3794f07ad8137ecf42 to finish bootstrapping
I20240503 08:11:46.095973 20910 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e7416e7c03a74a24b8fe2508111770d7 to finish bootstrapping
W20240503 08:11:46.096479 25213 tablet.cc:2369] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:46.104526 20910 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f9705fdde3854639ba85097a161f37e7 to finish bootstrapping
I20240503 08:11:46.112396 20910 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9d313001b0a647caba8f53540a2b67ac to finish bootstrapping
I20240503 08:11:46.120484 20910 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a0d221e07029452cae34b8920884507e to finish bootstrapping
I20240503 08:11:46.128055 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 24695
I20240503 08:11:46.129096 24817 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:46.210089 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 24695
I20240503 08:11:46.226161 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 24825
I20240503 08:11:46.227398 24947 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:46.310197 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 24825
I20240503 08:11:46.326205 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 24955
I20240503 08:11:46.327270 25077 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:46.407562 25082 heartbeater.cc:503] Master 127.20.107.190:44243 was elected leader, sending a full tablet report...
I20240503 08:11:46.408210 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 24955
W20240503 08:11:46.423725 24229 connection.cc:535] server connection from 127.20.107.134:46165 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:11:46.423955 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 25085
I20240503 08:11:46.425055 25207 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:46.511597 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 25085
I20240503 08:11:47.702180 25246 raft_consensus.cc:482] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:47.702376 25246 raft_consensus.cc:504] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:47.703745 25246 leader_election.cc:288] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d221e07029452cae34b8920884507e (127.20.107.135:45329)
W20240503 08:11:47.704834 24452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45329: connect: Connection refused (error 111)
W20240503 08:11:47.706353 24452 leader_election.cc:334] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a0d221e07029452cae34b8920884507e (127.20.107.135:45329): Network error: Client connection negotiation failed: client connection to 127.20.107.135:45329: connect: Connection refused (error 111)
I20240503 08:11:47.706535 24452 leader_election.cc:302] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6a640901590545eeb69e216d2399ea51; no voters: a0d221e07029452cae34b8920884507e
I20240503 08:11:47.706857 25246 raft_consensus.cc:2726] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:11:49.900846 25249 raft_consensus.cc:482] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:49.901041 25249 raft_consensus.cc:504] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:49.901543 25249 leader_election.cc:288] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d221e07029452cae34b8920884507e (127.20.107.135:45329)
W20240503 08:11:49.903859 24452 leader_election.cc:334] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a0d221e07029452cae34b8920884507e (127.20.107.135:45329): Network error: Client connection negotiation failed: client connection to 127.20.107.135:45329: connect: Connection refused (error 111)
I20240503 08:11:49.903995 24452 leader_election.cc:302] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6a640901590545eeb69e216d2399ea51; no voters: a0d221e07029452cae34b8920884507e
I20240503 08:11:49.904253 25249 raft_consensus.cc:2726] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:11:50.280231 24244 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58162:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:50.293740 24497 tablet_service.cc:1467] Processing CreateTablet for tablet 39ccbb3b60094ed1a8a09ae6b6c4e6a5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=492713985fb14342af075b33cdc4b9af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:50.294235 24497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ccbb3b60094ed1a8a09ae6b6c4e6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:50.300909 25260 tablet_bootstrap.cc:492] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51: Bootstrap starting.
I20240503 08:11:50.302801 25260 tablet_bootstrap.cc:654] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:50.304623 24365 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=492713985fb14342af075b33cdc4b9af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:50.304275 24626 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=492713985fb14342af075b33cdc4b9af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:50.305296 24627 tablet_service.cc:1467] Processing CreateTablet for tablet c85cf905ebfa4e7b91a0af9ed2fa77da (DEFAULT_TABLE table=rebalance_test_table_0 [id=492713985fb14342af075b33cdc4b9af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:50.305825 24365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:50.305847 24626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:50.304289 24364 tablet_service.cc:1467] Processing CreateTablet for tablet 39ccbb3b60094ed1a8a09ae6b6c4e6a5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=492713985fb14342af075b33cdc4b9af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:50.306571 24364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ccbb3b60094ed1a8a09ae6b6c4e6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:50.307359 24366 tablet_service.cc:1467] Processing CreateTablet for tablet c85cf905ebfa4e7b91a0af9ed2fa77da (DEFAULT_TABLE table=rebalance_test_table_0 [id=492713985fb14342af075b33cdc4b9af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:50.308017 24366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85cf905ebfa4e7b91a0af9ed2fa77da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:50.308830 24627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85cf905ebfa4e7b91a0af9ed2fa77da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:50.312163 25260 tablet_bootstrap.cc:492] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51: No bootstrap required, opened a new log
I20240503 08:11:50.312353 25260 ts_tablet_manager.cc:1367] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240503 08:11:50.313380 25260 raft_consensus.cc:348] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:50.313609 25260 raft_consensus.cc:374] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:50.313733 25260 raft_consensus.cc:729] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Initialized, Role: FOLLOWER
I20240503 08:11:50.313992 25260 consensus_queue.cc:260] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:50.315274 25260 ts_tablet_manager.cc:1398] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:50.328020 25263 tablet_bootstrap.cc:492] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42: Bootstrap starting.
I20240503 08:11:50.329289 25262 tablet_bootstrap.cc:492] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6: Bootstrap starting.
I20240503 08:11:50.332083 25263 tablet_bootstrap.cc:654] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:50.332674 25262 tablet_bootstrap.cc:654] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:50.333469 25263 log.cc:824] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:50.333897 25262 log.cc:824] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:50.335983 25263 tablet_bootstrap.cc:492] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42: No bootstrap required, opened a new log
I20240503 08:11:50.336212 25262 tablet_bootstrap.cc:492] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6: No bootstrap required, opened a new log
I20240503 08:11:50.336231 25263 ts_tablet_manager.cc:1367] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 08:11:50.336447 25262 ts_tablet_manager.cc:1367] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 08:11:50.338461 25249 raft_consensus.cc:482] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:50.338631 25249 raft_consensus.cc:504] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:50.339370 25249 leader_election.cc:288] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:50.343782 25262 raft_consensus.cc:348] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:50.344331 25262 raft_consensus.cc:374] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:50.344478 25262 raft_consensus.cc:729] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Initialized, Role: FOLLOWER
I20240503 08:11:50.344069 25263 raft_consensus.cc:348] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:50.344858 25263 raft_consensus.cc:374] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:50.345001 25263 raft_consensus.cc:729] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Initialized, Role: FOLLOWER
I20240503 08:11:50.345239 25262 consensus_queue.cc:260] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:50.345600 25263 consensus_queue.cc:260] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:50.347141 25262 ts_tablet_manager.cc:1398] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20240503 08:11:50.347995 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ccbb3b60094ed1a8a09ae6b6c4e6a5" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6" is_pre_election: true
I20240503 08:11:50.348856 25262 tablet_bootstrap.cc:492] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6: Bootstrap starting.
W20240503 08:11:50.349208 24450 leader_election.cc:341] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:50.349390 24450 leader_election.cc:302] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6a640901590545eeb69e216d2399ea51; no voters: b53502e2a1ae4c86a054d38048dd0df6
I20240503 08:11:50.349855 25249 raft_consensus.cc:2726] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:11:50.350595 25262 tablet_bootstrap.cc:654] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:50.352452 25262 tablet_bootstrap.cc:492] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6: No bootstrap required, opened a new log
I20240503 08:11:50.352594 25262 ts_tablet_manager.cc:1367] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:50.352977 25263 ts_tablet_manager.cc:1398] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20240503 08:11:50.353420 25262 raft_consensus.cc:348] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:50.353490 25263 tablet_bootstrap.cc:492] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42: Bootstrap starting.
I20240503 08:11:50.353569 25262 raft_consensus.cc:374] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:50.353705 25262 raft_consensus.cc:729] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Initialized, Role: FOLLOWER
I20240503 08:11:50.353948 25262 consensus_queue.cc:260] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:50.354326 25262 ts_tablet_manager.cc:1398] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:50.354550 25262 tablet_bootstrap.cc:492] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6: Bootstrap starting.
I20240503 08:11:50.355218 25263 tablet_bootstrap.cc:654] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:50.355780 25262 tablet_bootstrap.cc:654] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:50.357710 25263 tablet_bootstrap.cc:492] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42: No bootstrap required, opened a new log
I20240503 08:11:50.357879 25262 tablet_bootstrap.cc:492] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6: No bootstrap required, opened a new log
I20240503 08:11:50.357867 25263 ts_tablet_manager.cc:1367] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:50.358032 25262 ts_tablet_manager.cc:1367] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240503 08:11:50.358778 25262 raft_consensus.cc:348] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:50.358893 25263 raft_consensus.cc:348] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:50.359028 25262 raft_consensus.cc:374] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:50.359113 25263 raft_consensus.cc:374] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:50.359126 25262 raft_consensus.cc:729] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Initialized, Role: FOLLOWER
I20240503 08:11:50.359295 25263 raft_consensus.cc:729] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Initialized, Role: FOLLOWER
I20240503 08:11:50.359426 25262 consensus_queue.cc:260] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:50.359665 25263 consensus_queue.cc:260] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:50.360010 25262 ts_tablet_manager.cc:1398] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:50.360240 25263 ts_tablet_manager.cc:1398] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20240503 08:11:50.461180 24693 tablet.cc:2369] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:50.473336 25267 raft_consensus.cc:482] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:50.473604 25267 raft_consensus.cc:504] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:50.475203 25267 leader_election.cc:288] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:50.482208 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd5613d173f4b1eb0eacf2fec3a3ff0" candidate_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6" is_pre_election: true
I20240503 08:11:50.482578 24386 raft_consensus.cc:2454] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56fb8e40a4da4b3794f07ad8137ecf42 in term 0.
I20240503 08:11:50.483209 24580 leader_election.cc:302] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:50.483574 25267 raft_consensus.cc:2781] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:50.483739 25267 raft_consensus.cc:482] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:50.483840 25267 raft_consensus.cc:3037] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:50.486268 25267 raft_consensus.cc:504] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:50.486768 25267 leader_election.cc:288] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 election: Requested vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:50.487264 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd5613d173f4b1eb0eacf2fec3a3ff0" candidate_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
I20240503 08:11:50.487471 24386 raft_consensus.cc:3037] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:50.489596 24386 raft_consensus.cc:2454] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56fb8e40a4da4b3794f07ad8137ecf42 in term 1.
I20240503 08:11:50.489984 24580 leader_election.cc:302] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:50.490283 25267 raft_consensus.cc:2781] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:50.490737 25267 raft_consensus.cc:686] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Becoming Leader. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Running, Role: LEADER
I20240503 08:11:50.491286 25267 consensus_queue.cc:237] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:50.495831 24244 catalog_manager.cc:5484] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131). New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: HEALTHY } } }
W20240503 08:11:50.527065 24432 tablet.cc:2369] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:50.670549 25267 raft_consensus.cc:482] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:50.670809 25267 raft_consensus.cc:504] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:50.671322 25267 leader_election.cc:288] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:50.671901 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85cf905ebfa4e7b91a0af9ed2fa77da" candidate_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6" is_pre_election: true
I20240503 08:11:50.672116 24386 raft_consensus.cc:2454] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56fb8e40a4da4b3794f07ad8137ecf42 in term 0.
I20240503 08:11:50.672519 24580 leader_election.cc:302] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:50.672803 25267 raft_consensus.cc:2781] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:50.672917 25267 raft_consensus.cc:482] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:50.673030 25267 raft_consensus.cc:3037] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:50.674978 25267 raft_consensus.cc:504] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:50.675455 25267 leader_election.cc:288] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 election: Requested vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:50.675930 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85cf905ebfa4e7b91a0af9ed2fa77da" candidate_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
I20240503 08:11:50.676093 24386 raft_consensus.cc:3037] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:50.677994 24386 raft_consensus.cc:2454] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56fb8e40a4da4b3794f07ad8137ecf42 in term 1.
I20240503 08:11:50.678364 24580 leader_election.cc:302] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:50.678591 25267 raft_consensus.cc:2781] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:50.678756 25267 raft_consensus.cc:686] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Becoming Leader. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Running, Role: LEADER
I20240503 08:11:50.678987 25267 consensus_queue.cc:237] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:50.681331 24244 catalog_manager.cc:5484] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131). New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:50.841070 25266 raft_consensus.cc:482] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:50.841274 25266 raft_consensus.cc:504] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:50.843154 25266 leader_election.cc:288] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:11:50.863987 24517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ccbb3b60094ed1a8a09ae6b6c4e6a5" candidate_uuid: "b53502e2a1ae4c86a054d38048dd0df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a640901590545eeb69e216d2399ea51" is_pre_election: true
I20240503 08:11:50.864236 24517 raft_consensus.cc:2454] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b53502e2a1ae4c86a054d38048dd0df6 in term 0.
I20240503 08:11:50.864738 24320 leader_election.cc:302] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:50.865176 25266 raft_consensus.cc:2781] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:50.865326 25266 raft_consensus.cc:482] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:50.865417 25266 raft_consensus.cc:3037] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:50.867312 25266 raft_consensus.cc:504] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:50.867789 25266 leader_election.cc:288] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 1 election: Requested vote from peers 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:11:50.868322 24517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ccbb3b60094ed1a8a09ae6b6c4e6a5" candidate_uuid: "b53502e2a1ae4c86a054d38048dd0df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a640901590545eeb69e216d2399ea51"
I20240503 08:11:50.868525 24517 raft_consensus.cc:3037] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:50.870661 24517 raft_consensus.cc:2454] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b53502e2a1ae4c86a054d38048dd0df6 in term 1.
I20240503 08:11:50.871065 24320 leader_election.cc:302] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:50.871367 25266 raft_consensus.cc:2781] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:50.871737 25266 raft_consensus.cc:686] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 LEADER]: Becoming Leader. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Running, Role: LEADER
I20240503 08:11:50.872222 25266 consensus_queue.cc:237] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:50.876358 24244 catalog_manager.cc:5484] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129). New cstate: current_term: 1 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:50.888003 25267 consensus_queue.cc:1035] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:51.086027 25272 consensus_queue.cc:1035] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:51.338706 25282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:51.339087 25282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:51.346984 25282 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 08:11:51.417891 25266 consensus_queue.cc:1035] T 39ccbb3b60094ed1a8a09ae6b6c4e6a5 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 215.247 ms
time per row: 0.107624 ms
I20240503 08:11:51.706234 24244 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58162:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:51.714908 24627 tablet_service.cc:1467] Processing CreateTablet for tablet 61bf5a9f26034821b821a45ac37ba133 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bb0497226b9c4da186bef24f42f0762a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:51.715076 24497 tablet_service.cc:1467] Processing CreateTablet for tablet 61bf5a9f26034821b821a45ac37ba133 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bb0497226b9c4da186bef24f42f0762a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:51.715409 24627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61bf5a9f26034821b821a45ac37ba133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:51.715446 24497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61bf5a9f26034821b821a45ac37ba133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:51.715950 24496 tablet_service.cc:1467] Processing CreateTablet for tablet 7b858bcd6e1c469d9f19f142106908d3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bb0497226b9c4da186bef24f42f0762a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:51.716455 24496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b858bcd6e1c469d9f19f142106908d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:51.716003 24495 tablet_service.cc:1467] Processing CreateTablet for tablet ca9558ccb5444f61b3ee86625b2f1097 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bb0497226b9c4da186bef24f42f0762a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:51.719259 24495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca9558ccb5444f61b3ee86625b2f1097. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:51.720794 24366 tablet_service.cc:1467] Processing CreateTablet for tablet ca9558ccb5444f61b3ee86625b2f1097 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bb0497226b9c4da186bef24f42f0762a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:51.721302 24366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca9558ccb5444f61b3ee86625b2f1097. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:51.721674 24626 tablet_service.cc:1467] Processing CreateTablet for tablet 7b858bcd6e1c469d9f19f142106908d3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bb0497226b9c4da186bef24f42f0762a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:51.722039 24626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b858bcd6e1c469d9f19f142106908d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:51.728642 25337 tablet_bootstrap.cc:492] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51: Bootstrap starting.
I20240503 08:11:51.730026 25338 tablet_bootstrap.cc:492] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42: Bootstrap starting.
I20240503 08:11:51.730891 25339 tablet_bootstrap.cc:492] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6: Bootstrap starting.
I20240503 08:11:51.731832 25338 tablet_bootstrap.cc:654] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:51.732273 25337 tablet_bootstrap.cc:654] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:51.732317 25339 tablet_bootstrap.cc:654] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:51.734022 25338 tablet_bootstrap.cc:492] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42: No bootstrap required, opened a new log
I20240503 08:11:51.734163 25338 ts_tablet_manager.cc:1367] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:51.734295 25339 tablet_bootstrap.cc:492] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6: No bootstrap required, opened a new log
I20240503 08:11:51.734431 25337 tablet_bootstrap.cc:492] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51: No bootstrap required, opened a new log
I20240503 08:11:51.734445 25339 ts_tablet_manager.cc:1367] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:51.734575 25337 ts_tablet_manager.cc:1367] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 08:11:51.734907 25338 raft_consensus.cc:348] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:51.735194 25338 raft_consensus.cc:374] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:51.735198 25339 raft_consensus.cc:348] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:51.735343 25338 raft_consensus.cc:729] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.735271 25337 raft_consensus.cc:348] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:51.735406 25339 raft_consensus.cc:374] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:51.735450 25337 raft_consensus.cc:374] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:51.735492 25339 raft_consensus.cc:729] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.735538 25337 raft_consensus.cc:729] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.735605 25338 consensus_queue.cc:260] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:51.735713 25339 consensus_queue.cc:260] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:51.735785 25337 consensus_queue.cc:260] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:51.736177 25338 ts_tablet_manager.cc:1398] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:51.736277 25339 ts_tablet_manager.cc:1398] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:51.736411 25338 tablet_bootstrap.cc:492] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42: Bootstrap starting.
I20240503 08:11:51.736946 25337 ts_tablet_manager.cc:1398] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:51.737216 25337 tablet_bootstrap.cc:492] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51: Bootstrap starting.
I20240503 08:11:51.738181 25338 tablet_bootstrap.cc:654] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:51.739446 25337 tablet_bootstrap.cc:654] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:51.742964 25338 tablet_bootstrap.cc:492] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42: No bootstrap required, opened a new log
I20240503 08:11:51.743196 25338 ts_tablet_manager.cc:1367] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240503 08:11:51.743839 25338 raft_consensus.cc:348] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:51.744016 25338 raft_consensus.cc:374] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:51.744103 25338 raft_consensus.cc:729] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.744403 25338 consensus_queue.cc:260] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:51.745055 25338 ts_tablet_manager.cc:1398] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:51.745321 25337 tablet_bootstrap.cc:492] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51: No bootstrap required, opened a new log
I20240503 08:11:51.745533 25337 ts_tablet_manager.cc:1367] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20240503 08:11:51.746377 25337 raft_consensus.cc:348] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:51.746650 25337 raft_consensus.cc:374] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:51.746726 25337 raft_consensus.cc:729] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.746951 25337 consensus_queue.cc:260] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:51.747800 25337 ts_tablet_manager.cc:1398] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:51.748143 25337 tablet_bootstrap.cc:492] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51: Bootstrap starting.
I20240503 08:11:51.749708 25337 tablet_bootstrap.cc:654] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:51.751586 25337 tablet_bootstrap.cc:492] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51: No bootstrap required, opened a new log
I20240503 08:11:51.751721 25337 ts_tablet_manager.cc:1367] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:51.752341 25337 raft_consensus.cc:348] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:51.752575 25337 raft_consensus.cc:374] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:51.752667 25337 raft_consensus.cc:729] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.752936 25337 consensus_queue.cc:260] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:51.753361 25337 ts_tablet_manager.cc:1398] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:51.900157 25323 raft_consensus.cc:482] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:51.900353 25323 raft_consensus.cc:504] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:51.901064 25323 leader_election.cc:288] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:11:51.907208 24517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b858bcd6e1c469d9f19f142106908d3" candidate_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a640901590545eeb69e216d2399ea51" is_pre_election: true
I20240503 08:11:51.907423 24517 raft_consensus.cc:2454] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56fb8e40a4da4b3794f07ad8137ecf42 in term 0.
I20240503 08:11:51.907809 24581 leader_election.cc:302] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, 6a640901590545eeb69e216d2399ea51; no voters:
I20240503 08:11:51.908067 25323 raft_consensus.cc:2781] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:51.908179 25323 raft_consensus.cc:482] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:51.908258 25323 raft_consensus.cc:3037] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:51.910108 25323 raft_consensus.cc:504] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:51.910529 25323 leader_election.cc:288] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 election: Requested vote from peers 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:11:51.911039 24517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b858bcd6e1c469d9f19f142106908d3" candidate_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a640901590545eeb69e216d2399ea51"
I20240503 08:11:51.911196 24517 raft_consensus.cc:3037] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:51.913004 24517 raft_consensus.cc:2454] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56fb8e40a4da4b3794f07ad8137ecf42 in term 1.
I20240503 08:11:51.913331 24581 leader_election.cc:302] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, 6a640901590545eeb69e216d2399ea51; no voters:
I20240503 08:11:51.913594 25323 raft_consensus.cc:2781] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:51.913733 25323 raft_consensus.cc:686] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Becoming Leader. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Running, Role: LEADER
I20240503 08:11:51.913992 25323 consensus_queue.cc:237] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:51.916494 24244 catalog_manager.cc:5484] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131). New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:52.014791 25306 raft_consensus.cc:482] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:52.014994 25306 raft_consensus.cc:504] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:52.015632 25306 leader_election.cc:288] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:52.016243 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9558ccb5444f61b3ee86625b2f1097" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6" is_pre_election: true
I20240503 08:11:52.016505 24386 raft_consensus.cc:2454] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a640901590545eeb69e216d2399ea51 in term 0.
I20240503 08:11:52.017014 24450 leader_election.cc:302] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:52.017272 25306 raft_consensus.cc:2781] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:52.017381 25306 raft_consensus.cc:482] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:52.017450 25306 raft_consensus.cc:3037] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:52.019557 25306 raft_consensus.cc:504] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:52.020038 25306 leader_election.cc:288] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 election: Requested vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:52.020517 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9558ccb5444f61b3ee86625b2f1097" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
I20240503 08:11:52.020757 24386 raft_consensus.cc:3037] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:52.023561 24386 raft_consensus.cc:2454] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a640901590545eeb69e216d2399ea51 in term 1.
I20240503 08:11:52.023972 24450 leader_election.cc:302] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:52.024258 25306 raft_consensus.cc:2781] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:52.024667 25306 raft_consensus.cc:686] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Becoming Leader. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Running, Role: LEADER
I20240503 08:11:52.025149 25306 consensus_queue.cc:237] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:52.029114 24244 catalog_manager.cc:5484] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a640901590545eeb69e216d2399ea51 (127.20.107.130). New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:52.125072 25306 raft_consensus.cc:482] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:52.125283 25306 raft_consensus.cc:504] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:52.125856 25306 leader_election.cc:288] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:11:52.131546 24647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61bf5a9f26034821b821a45ac37ba133" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" is_pre_election: true
I20240503 08:11:52.131857 24647 raft_consensus.cc:2454] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a640901590545eeb69e216d2399ea51 in term 0.
I20240503 08:11:52.132248 24451 leader_election.cc:302] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, 6a640901590545eeb69e216d2399ea51; no voters:
I20240503 08:11:52.132548 25306 raft_consensus.cc:2781] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:52.132633 25306 raft_consensus.cc:482] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:52.132709 25306 raft_consensus.cc:3037] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:52.134541 25306 raft_consensus.cc:504] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:52.134995 25306 leader_election.cc:288] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 election: Requested vote from peers 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:11:52.135449 24647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61bf5a9f26034821b821a45ac37ba133" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42"
I20240503 08:11:52.135603 24647 raft_consensus.cc:3037] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:52.137465 24647 raft_consensus.cc:2454] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a640901590545eeb69e216d2399ea51 in term 1.
I20240503 08:11:52.137833 24451 leader_election.cc:302] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, 6a640901590545eeb69e216d2399ea51; no voters:
I20240503 08:11:52.138068 25306 raft_consensus.cc:2781] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:52.138208 25306 raft_consensus.cc:686] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Becoming Leader. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Running, Role: LEADER
I20240503 08:11:52.138447 25306 consensus_queue.cc:237] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:52.140638 24244 catalog_manager.cc:5484] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a640901590545eeb69e216d2399ea51 (127.20.107.130). New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:52.368018 25306 raft_consensus.cc:482] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:52.368242 25306 raft_consensus.cc:504] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:52.368968 25306 leader_election.cc:288] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d221e07029452cae34b8920884507e (127.20.107.135:45329)
W20240503 08:11:52.370677 24452 leader_election.cc:334] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a0d221e07029452cae34b8920884507e (127.20.107.135:45329): Network error: Client connection negotiation failed: client connection to 127.20.107.135:45329: connect: Connection refused (error 111)
I20240503 08:11:52.370846 24452 leader_election.cc:302] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6a640901590545eeb69e216d2399ea51; no voters: a0d221e07029452cae34b8920884507e
I20240503 08:11:52.371129 25306 raft_consensus.cc:2726] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:11:52.450670 25323 consensus_queue.cc:1035] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:52.455271 25306 consensus_queue.cc:1035] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:52.537315 25354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:52.537647 25354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:52.545090 25354 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 08:11:52.577383 25306 consensus_queue.cc:1035] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 580.001 ms
time per row: 0.290001 ms
I20240503 08:11:53.267374 24243 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58162:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:53.277128 24626 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2772d08c06498db2cb0b02497ea042 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f8b8cf1869a436f9c012044eb10dc54]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:53.277285 24366 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2772d08c06498db2cb0b02497ea042 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f8b8cf1869a436f9c012044eb10dc54]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:53.277642 24626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2772d08c06498db2cb0b02497ea042. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:53.277714 24366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2772d08c06498db2cb0b02497ea042. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:53.278354 24627 tablet_service.cc:1467] Processing CreateTablet for tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f8b8cf1869a436f9c012044eb10dc54]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:53.278390 24497 tablet_service.cc:1467] Processing CreateTablet for tablet 52edf3b518be4f45bb347dbbe4bf7fd9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f8b8cf1869a436f9c012044eb10dc54]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:53.278700 24627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:53.278805 24497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52edf3b518be4f45bb347dbbe4bf7fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:53.281780 24365 tablet_service.cc:1467] Processing CreateTablet for tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f8b8cf1869a436f9c012044eb10dc54]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:53.282163 24365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:53.285512 24364 tablet_service.cc:1467] Processing CreateTablet for tablet 52edf3b518be4f45bb347dbbe4bf7fd9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f8b8cf1869a436f9c012044eb10dc54]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:53.286000 24364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52edf3b518be4f45bb347dbbe4bf7fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:53.290958 25411 tablet_bootstrap.cc:492] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42: Bootstrap starting.
I20240503 08:11:53.293920 25411 tablet_bootstrap.cc:654] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:53.297256 25413 tablet_bootstrap.cc:492] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6: Bootstrap starting.
I20240503 08:11:53.297362 25412 tablet_bootstrap.cc:492] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51: Bootstrap starting.
I20240503 08:11:53.298951 25413 tablet_bootstrap.cc:654] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:53.299289 25412 tablet_bootstrap.cc:654] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:53.300783 25413 tablet_bootstrap.cc:492] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6: No bootstrap required, opened a new log
I20240503 08:11:53.300966 25413 ts_tablet_manager.cc:1367] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:53.301635 25411 tablet_bootstrap.cc:492] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42: No bootstrap required, opened a new log
I20240503 08:11:53.301638 25412 tablet_bootstrap.cc:492] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51: No bootstrap required, opened a new log
I20240503 08:11:53.301597 25413 raft_consensus.cc:348] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:53.301774 25413 raft_consensus.cc:374] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:53.301784 25411 ts_tablet_manager.cc:1367] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20240503 08:11:53.301788 25412 ts_tablet_manager.cc:1367] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 08:11:53.301877 25413 raft_consensus.cc:729] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Initialized, Role: FOLLOWER
I20240503 08:11:53.302116 25413 consensus_queue.cc:260] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:53.302590 25413 ts_tablet_manager.cc:1398] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:11:53.302508 25411 raft_consensus.cc:348] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:53.302634 25412 raft_consensus.cc:348] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:53.302790 25411 raft_consensus.cc:374] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:53.302845 25412 raft_consensus.cc:374] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:53.302906 25413 tablet_bootstrap.cc:492] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6: Bootstrap starting.
I20240503 08:11:53.302956 25411 raft_consensus.cc:729] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Initialized, Role: FOLLOWER
I20240503 08:11:53.302981 25412 raft_consensus.cc:729] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Initialized, Role: FOLLOWER
I20240503 08:11:53.303234 25412 consensus_queue.cc:260] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:53.303287 25411 consensus_queue.cc:260] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:53.304780 25412 ts_tablet_manager.cc:1398] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:11:53.305344 25411 ts_tablet_manager.cc:1398] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:11:53.305646 25411 tablet_bootstrap.cc:492] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42: Bootstrap starting.
I20240503 08:11:53.306582 25397 raft_consensus.cc:482] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:53.306744 25397 raft_consensus.cc:504] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:53.306911 25413 tablet_bootstrap.cc:654] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:53.307175 25411 tablet_bootstrap.cc:654] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:53.307310 25397 leader_election.cc:288] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:53.309162 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52edf3b518be4f45bb347dbbe4bf7fd9" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6" is_pre_election: true
W20240503 08:11:53.309808 24450 leader_election.cc:341] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:53.309938 24450 leader_election.cc:302] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6a640901590545eeb69e216d2399ea51; no voters: b53502e2a1ae4c86a054d38048dd0df6
I20240503 08:11:53.310303 25397 raft_consensus.cc:2726] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:11:53.310863 25411 tablet_bootstrap.cc:492] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42: No bootstrap required, opened a new log
I20240503 08:11:53.311010 25411 ts_tablet_manager.cc:1367] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 08:11:53.311651 25411 raft_consensus.cc:348] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:53.311811 25411 raft_consensus.cc:374] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:53.311884 25411 raft_consensus.cc:729] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Initialized, Role: FOLLOWER
I20240503 08:11:53.312119 25411 consensus_queue.cc:260] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:53.312564 25411 ts_tablet_manager.cc:1398] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:11:53.313815 25413 tablet_bootstrap.cc:492] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6: No bootstrap required, opened a new log
I20240503 08:11:53.313951 25413 ts_tablet_manager.cc:1367] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240503 08:11:53.314579 25413 raft_consensus.cc:348] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:53.314755 25413 raft_consensus.cc:374] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:53.314863 25413 raft_consensus.cc:729] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Initialized, Role: FOLLOWER
I20240503 08:11:53.315119 25413 consensus_queue.cc:260] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:53.315609 25413 ts_tablet_manager.cc:1398] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:53.315922 25413 tablet_bootstrap.cc:492] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6: Bootstrap starting.
I20240503 08:11:53.317263 25413 tablet_bootstrap.cc:654] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:53.319128 25413 tablet_bootstrap.cc:492] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6: No bootstrap required, opened a new log
I20240503 08:11:53.319262 25413 ts_tablet_manager.cc:1367] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:53.319958 25413 raft_consensus.cc:348] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:53.320094 25413 raft_consensus.cc:374] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:53.320159 25413 raft_consensus.cc:729] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Initialized, Role: FOLLOWER
I20240503 08:11:53.320333 25413 consensus_queue.cc:260] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:53.320749 25413 ts_tablet_manager.cc:1398] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:53.419214 25357 raft_consensus.cc:482] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:53.419399 25357 raft_consensus.cc:504] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:53.420025 25357 leader_election.cc:288] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:53.420600 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08df2ee2ca344cd2abd5ecb67ba9ffc1" candidate_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6" is_pre_election: true
I20240503 08:11:53.420831 24386 raft_consensus.cc:2454] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56fb8e40a4da4b3794f07ad8137ecf42 in term 0.
I20240503 08:11:53.421299 24580 leader_election.cc:302] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:53.421567 25357 raft_consensus.cc:2781] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:53.421679 25357 raft_consensus.cc:482] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:53.421785 25357 raft_consensus.cc:3037] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:53.423633 25357 raft_consensus.cc:504] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:53.424044 25357 leader_election.cc:288] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 election: Requested vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:53.424494 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08df2ee2ca344cd2abd5ecb67ba9ffc1" candidate_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
I20240503 08:11:53.424716 24386 raft_consensus.cc:3037] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:53.426512 24386 raft_consensus.cc:2454] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56fb8e40a4da4b3794f07ad8137ecf42 in term 1.
I20240503 08:11:53.426877 24580 leader_election.cc:302] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:53.427138 25357 raft_consensus.cc:2781] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:53.427495 25357 raft_consensus.cc:686] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Becoming Leader. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Running, Role: LEADER
I20240503 08:11:53.427803 25357 consensus_queue.cc:237] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:53.430092 24244 catalog_manager.cc:5484] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131). New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:53.463400 25405 raft_consensus.cc:482] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:53.463593 25405 raft_consensus.cc:504] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:53.464187 25405 leader_election.cc:288] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:11:53.464835 24517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52edf3b518be4f45bb347dbbe4bf7fd9" candidate_uuid: "b53502e2a1ae4c86a054d38048dd0df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a640901590545eeb69e216d2399ea51" is_pre_election: true
I20240503 08:11:53.465123 24517 raft_consensus.cc:2454] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b53502e2a1ae4c86a054d38048dd0df6 in term 0.
I20240503 08:11:53.465588 24320 leader_election.cc:302] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:53.465896 25405 raft_consensus.cc:2781] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:53.466006 25405 raft_consensus.cc:482] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:53.466101 25405 raft_consensus.cc:3037] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:53.468189 25405 raft_consensus.cc:504] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:53.468734 25405 leader_election.cc:288] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 1 election: Requested vote from peers 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:11:53.469344 24517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52edf3b518be4f45bb347dbbe4bf7fd9" candidate_uuid: "b53502e2a1ae4c86a054d38048dd0df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a640901590545eeb69e216d2399ea51"
I20240503 08:11:53.469555 24517 raft_consensus.cc:3037] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:53.471563 24517 raft_consensus.cc:2454] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b53502e2a1ae4c86a054d38048dd0df6 in term 1.
I20240503 08:11:53.472018 24320 leader_election.cc:302] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:53.472339 25405 raft_consensus.cc:2781] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:53.472815 25405 raft_consensus.cc:686] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 LEADER]: Becoming Leader. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Running, Role: LEADER
I20240503 08:11:53.473165 25405 consensus_queue.cc:237] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:53.476073 24244 catalog_manager.cc:5484] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129). New cstate: current_term: 1 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:53.754873 25357 raft_consensus.cc:482] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:53.755107 25357 raft_consensus.cc:504] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:53.755630 25357 leader_election.cc:288] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:53.756261 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2772d08c06498db2cb0b02497ea042" candidate_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6" is_pre_election: true
I20240503 08:11:53.756460 24386 raft_consensus.cc:2454] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56fb8e40a4da4b3794f07ad8137ecf42 in term 0.
I20240503 08:11:53.756911 24580 leader_election.cc:302] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:53.757189 25357 raft_consensus.cc:2781] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:53.757284 25357 raft_consensus.cc:482] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:53.757376 25357 raft_consensus.cc:3037] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:53.759202 25357 raft_consensus.cc:504] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:53.759795 25357 leader_election.cc:288] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 election: Requested vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:53.760387 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2772d08c06498db2cb0b02497ea042" candidate_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
I20240503 08:11:53.760627 24386 raft_consensus.cc:3037] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:53.762658 24386 raft_consensus.cc:2454] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56fb8e40a4da4b3794f07ad8137ecf42 in term 1.
I20240503 08:11:53.763089 24580 leader_election.cc:302] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:53.763388 25357 raft_consensus.cc:2781] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:53.763546 25357 raft_consensus.cc:686] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Becoming Leader. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Running, Role: LEADER
I20240503 08:11:53.763792 25357 consensus_queue.cc:237] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } }
I20240503 08:11:53.766284 24244 catalog_manager.cc:5484] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131). New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:54.011252 25357 consensus_queue.cc:1035] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:54.025676 25405 consensus_queue.cc:1035] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:54.164307 25420 consensus_queue.cc:1035] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:54.210484 25425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:54.210812 25425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:54.218320 25425 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 178.347 ms
time per row: 0.0891734 ms
I20240503 08:11:54.547744 24241 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58162:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:54.556222 24626 tablet_service.cc:1467] Processing CreateTablet for tablet 3071edf0c1004a6496d1cb46591d422e (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6ccc802e346493aa6739ba0f901f14d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:54.556376 24497 tablet_service.cc:1467] Processing CreateTablet for tablet 3071edf0c1004a6496d1cb46591d422e (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6ccc802e346493aa6739ba0f901f14d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:54.556809 24497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3071edf0c1004a6496d1cb46591d422e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:54.556859 24626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3071edf0c1004a6496d1cb46591d422e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:54.557215 24365 tablet_service.cc:1467] Processing CreateTablet for tablet 3c65da71faf34351885b5033d3d68835 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6ccc802e346493aa6739ba0f901f14d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:54.557590 24365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c65da71faf34351885b5033d3d68835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:54.563470 24496 tablet_service.cc:1467] Processing CreateTablet for tablet e0756a24eb5d4e26b2d44e46638a74be (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6ccc802e346493aa6739ba0f901f14d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:54.563431 24627 tablet_service.cc:1467] Processing CreateTablet for tablet e0756a24eb5d4e26b2d44e46638a74be (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6ccc802e346493aa6739ba0f901f14d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:54.563951 24495 tablet_service.cc:1467] Processing CreateTablet for tablet 3c65da71faf34351885b5033d3d68835 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6ccc802e346493aa6739ba0f901f14d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:54.564061 24627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0756a24eb5d4e26b2d44e46638a74be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:54.564090 24496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0756a24eb5d4e26b2d44e46638a74be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:54.564316 24495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c65da71faf34351885b5033d3d68835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:54.568219 25480 tablet_bootstrap.cc:492] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6: Bootstrap starting.
I20240503 08:11:54.570127 25480 tablet_bootstrap.cc:654] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:54.571835 25481 tablet_bootstrap.cc:492] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51: Bootstrap starting.
I20240503 08:11:54.574527 25481 tablet_bootstrap.cc:654] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:54.580192 25483 tablet_bootstrap.cc:492] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42: Bootstrap starting.
I20240503 08:11:54.581667 25480 tablet_bootstrap.cc:492] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6: No bootstrap required, opened a new log
I20240503 08:11:54.581662 25481 tablet_bootstrap.cc:492] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51: No bootstrap required, opened a new log
I20240503 08:11:54.581812 25481 ts_tablet_manager.cc:1367] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240503 08:11:54.581864 25480 ts_tablet_manager.cc:1367] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.005s
I20240503 08:11:54.582315 25483 tablet_bootstrap.cc:654] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:54.582535 25480 raft_consensus.cc:348] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.582700 25480 raft_consensus.cc:374] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:54.582664 25481 raft_consensus.cc:348] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.582826 25480 raft_consensus.cc:729] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Initialized, Role: FOLLOWER
I20240503 08:11:54.582893 25481 raft_consensus.cc:374] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:54.583003 25481 raft_consensus.cc:729] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Initialized, Role: FOLLOWER
I20240503 08:11:54.583076 25480 consensus_queue.cc:260] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.583724 25480 ts_tablet_manager.cc:1398] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:54.583245 25481 consensus_queue.cc:260] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.584465 25481 ts_tablet_manager.cc:1398] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:54.584756 25481 tablet_bootstrap.cc:492] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51: Bootstrap starting.
I20240503 08:11:54.585847 25483 tablet_bootstrap.cc:492] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42: No bootstrap required, opened a new log
I20240503 08:11:54.586001 25483 ts_tablet_manager.cc:1367] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:11:54.586349 25481 tablet_bootstrap.cc:654] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:54.586827 25483 raft_consensus.cc:348] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.587098 25483 raft_consensus.cc:374] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:54.587234 25483 raft_consensus.cc:729] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Initialized, Role: FOLLOWER
I20240503 08:11:54.587554 25483 consensus_queue.cc:260] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.588310 25483 ts_tablet_manager.cc:1398] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:54.589107 25483 tablet_bootstrap.cc:492] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42: Bootstrap starting.
I20240503 08:11:54.589210 25481 tablet_bootstrap.cc:492] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51: No bootstrap required, opened a new log
I20240503 08:11:54.589390 25481 ts_tablet_manager.cc:1367] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:11:54.590355 25483 tablet_bootstrap.cc:654] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:54.590579 25481 raft_consensus.cc:348] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.590777 25481 raft_consensus.cc:374] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:54.590863 25481 raft_consensus.cc:729] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Initialized, Role: FOLLOWER
I20240503 08:11:54.591091 25481 consensus_queue.cc:260] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.591694 25481 ts_tablet_manager.cc:1398] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:54.592345 25483 tablet_bootstrap.cc:492] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42: No bootstrap required, opened a new log
I20240503 08:11:54.592551 25483 ts_tablet_manager.cc:1367] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:54.592782 25481 tablet_bootstrap.cc:492] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51: Bootstrap starting.
I20240503 08:11:54.593468 25483 raft_consensus.cc:348] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.593637 25483 raft_consensus.cc:374] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:54.593715 25483 raft_consensus.cc:729] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Initialized, Role: FOLLOWER
I20240503 08:11:54.593942 25483 consensus_queue.cc:260] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.594219 25481 tablet_bootstrap.cc:654] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:54.594448 25483 ts_tablet_manager.cc:1398] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:11:54.596599 25481 tablet_bootstrap.cc:492] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51: No bootstrap required, opened a new log
I20240503 08:11:54.596710 25481 ts_tablet_manager.cc:1367] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:54.597385 25481 raft_consensus.cc:348] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.597530 25481 raft_consensus.cc:374] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:54.597620 25481 raft_consensus.cc:729] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Initialized, Role: FOLLOWER
I20240503 08:11:54.597847 25481 consensus_queue.cc:260] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.598330 25481 ts_tablet_manager.cc:1398] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:11:54.667300 25397 raft_consensus.cc:482] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:54.667515 25397 raft_consensus.cc:504] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.668005 25397 leader_election.cc:288] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:54.668623 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c65da71faf34351885b5033d3d68835" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6" is_pre_election: true
I20240503 08:11:54.668859 24386 raft_consensus.cc:2454] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a640901590545eeb69e216d2399ea51 in term 0.
I20240503 08:11:54.669359 24450 leader_election.cc:302] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:54.669636 25397 raft_consensus.cc:2781] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:54.669746 25397 raft_consensus.cc:482] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:54.669829 25397 raft_consensus.cc:3037] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:54.671792 25397 raft_consensus.cc:504] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.672268 25397 leader_election.cc:288] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 election: Requested vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:54.672919 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c65da71faf34351885b5033d3d68835" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
I20240503 08:11:54.673135 24386 raft_consensus.cc:3037] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:54.675135 24386 raft_consensus.cc:2454] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a640901590545eeb69e216d2399ea51 in term 1.
I20240503 08:11:54.675616 24450 leader_election.cc:302] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:54.675935 25397 raft_consensus.cc:2781] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:54.676276 25397 raft_consensus.cc:686] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Becoming Leader. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Running, Role: LEADER
I20240503 08:11:54.676595 25397 consensus_queue.cc:237] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.679071 24236 catalog_manager.cc:5484] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a640901590545eeb69e216d2399ea51 (127.20.107.130). New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:54.681550 25397 raft_consensus.cc:482] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:54.681710 25397 raft_consensus.cc:504] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.682355 25397 leader_election.cc:288] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:11:54.682757 24646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3071edf0c1004a6496d1cb46591d422e" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" is_pre_election: true
I20240503 08:11:54.682945 24646 raft_consensus.cc:2454] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a640901590545eeb69e216d2399ea51 in term 0.
I20240503 08:11:54.683315 24451 leader_election.cc:302] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, 6a640901590545eeb69e216d2399ea51; no voters:
I20240503 08:11:54.683575 25397 raft_consensus.cc:2781] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:54.683661 25397 raft_consensus.cc:482] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:54.683779 25397 raft_consensus.cc:3037] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:54.685819 25397 raft_consensus.cc:504] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.686219 25397 leader_election.cc:288] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 election: Requested vote from peers 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:11:54.686614 24646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3071edf0c1004a6496d1cb46591d422e" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42"
I20240503 08:11:54.686785 24646 raft_consensus.cc:3037] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:54.688635 24646 raft_consensus.cc:2454] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a640901590545eeb69e216d2399ea51 in term 1.
I20240503 08:11:54.689016 24451 leader_election.cc:302] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, 6a640901590545eeb69e216d2399ea51; no voters:
I20240503 08:11:54.689282 25397 raft_consensus.cc:2781] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:54.689435 25397 raft_consensus.cc:686] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Becoming Leader. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Running, Role: LEADER
I20240503 08:11:54.689692 25397 consensus_queue.cc:237] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.691905 24236 catalog_manager.cc:5484] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a640901590545eeb69e216d2399ea51 (127.20.107.130). New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:54.766732 25397 raft_consensus.cc:482] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:54.766966 25397 raft_consensus.cc:504] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.767500 25397 leader_election.cc:288] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:11:54.768110 24646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0756a24eb5d4e26b2d44e46638a74be" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" is_pre_election: true
I20240503 08:11:54.768337 24646 raft_consensus.cc:2454] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a640901590545eeb69e216d2399ea51 in term 0.
I20240503 08:11:54.768757 24451 leader_election.cc:302] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, 6a640901590545eeb69e216d2399ea51; no voters:
I20240503 08:11:54.769037 25397 raft_consensus.cc:2781] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:54.769142 25397 raft_consensus.cc:482] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:54.769225 25397 raft_consensus.cc:3037] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:54.771090 25397 raft_consensus.cc:504] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.771581 25397 leader_election.cc:288] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 election: Requested vote from peers 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:11:54.772154 24646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0756a24eb5d4e26b2d44e46638a74be" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42"
I20240503 08:11:54.772358 24646 raft_consensus.cc:3037] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:54.774762 24646 raft_consensus.cc:2454] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a640901590545eeb69e216d2399ea51 in term 1.
I20240503 08:11:54.775166 24451 leader_election.cc:302] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, 6a640901590545eeb69e216d2399ea51; no voters:
I20240503 08:11:54.775463 25397 raft_consensus.cc:2781] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:54.775619 25397 raft_consensus.cc:686] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Becoming Leader. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Running, Role: LEADER
I20240503 08:11:54.775892 25397 consensus_queue.cc:237] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:54.778703 24236 catalog_manager.cc:5484] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a640901590545eeb69e216d2399ea51 (127.20.107.130). New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } }
W20240503 08:11:55.150123 25493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:55.150501 25493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:55.158272 25493 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 08:11:55.240939 25397 consensus_queue.cc:1035] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:55.276474 25397 consensus_queue.cc:1035] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:55.321568 24646 raft_consensus.cc:1261] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:55.329113 25529 consensus_queue.cc:1035] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 08:11:55.365272 25517 mvcc.cc:205] Tried to move back new op lower bound from 7023509157125095424 to 7023509154923610112. Current Snapshot: MvccSnapshot[applied={T|T < 7023509157082681344}]
I20240503 08:11:55.393803 25518 mvcc.cc:205] Tried to move back new op lower bound from 7023509157125095424 to 7023509154923610112. Current Snapshot: MvccSnapshot[applied={T|T < 7023509157082681344}]
INSERT report
rows total: 2000
time total: 189.907 ms
time per row: 0.0949534 ms
I20240503 08:11:55.490006 24243 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58162:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:55.499289 24627 tablet_service.cc:1467] Processing CreateTablet for tablet 83697ce9030f4981af717d18eb6757a4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=72d93aae96774cf887b14b98c748dffc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:55.499519 24626 tablet_service.cc:1467] Processing CreateTablet for tablet d73b0d5398cd484ba45515cf95cd5b14 (DEFAULT_TABLE table=rebalance_test_table_4 [id=72d93aae96774cf887b14b98c748dffc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:55.499776 24627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83697ce9030f4981af717d18eb6757a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:55.500072 24365 tablet_service.cc:1467] Processing CreateTablet for tablet 83697ce9030f4981af717d18eb6757a4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=72d93aae96774cf887b14b98c748dffc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:55.500172 24364 tablet_service.cc:1467] Processing CreateTablet for tablet d8a32af7adaf410582e1b898257db62d (DEFAULT_TABLE table=rebalance_test_table_4 [id=72d93aae96774cf887b14b98c748dffc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:55.500504 24365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83697ce9030f4981af717d18eb6757a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:55.500510 24496 tablet_service.cc:1467] Processing CreateTablet for tablet d8a32af7adaf410582e1b898257db62d (DEFAULT_TABLE table=rebalance_test_table_4 [id=72d93aae96774cf887b14b98c748dffc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:55.500694 24364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a32af7adaf410582e1b898257db62d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:55.500919 24496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a32af7adaf410582e1b898257db62d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:55.501425 24366 tablet_service.cc:1467] Processing CreateTablet for tablet d73b0d5398cd484ba45515cf95cd5b14 (DEFAULT_TABLE table=rebalance_test_table_4 [id=72d93aae96774cf887b14b98c748dffc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:55.501799 24366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73b0d5398cd484ba45515cf95cd5b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:55.505252 24626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73b0d5398cd484ba45515cf95cd5b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:55.524960 25545 tablet_bootstrap.cc:492] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51: Bootstrap starting.
I20240503 08:11:55.527004 25544 tablet_bootstrap.cc:492] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42: Bootstrap starting.
I20240503 08:11:55.527592 25545 tablet_bootstrap.cc:654] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:55.528478 25544 tablet_bootstrap.cc:654] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:55.529341 25543 tablet_bootstrap.cc:492] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6: Bootstrap starting.
I20240503 08:11:55.531349 25543 tablet_bootstrap.cc:654] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:55.533914 25543 tablet_bootstrap.cc:492] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6: No bootstrap required, opened a new log
I20240503 08:11:55.534112 25543 ts_tablet_manager.cc:1367] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:11:55.535173 25543 raft_consensus.cc:348] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:55.535454 25543 raft_consensus.cc:374] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:55.535624 25543 raft_consensus.cc:729] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Initialized, Role: FOLLOWER
I20240503 08:11:55.535953 25543 consensus_queue.cc:260] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:55.536633 25543 ts_tablet_manager.cc:1398] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:55.537017 25543 tablet_bootstrap.cc:492] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6: Bootstrap starting.
I20240503 08:11:55.537417 25544 tablet_bootstrap.cc:492] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42: No bootstrap required, opened a new log
I20240503 08:11:55.537631 25544 ts_tablet_manager.cc:1367] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240503 08:11:55.537678 25545 tablet_bootstrap.cc:492] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51: No bootstrap required, opened a new log
I20240503 08:11:55.537837 25545 ts_tablet_manager.cc:1367] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240503 08:11:55.538581 25543 tablet_bootstrap.cc:654] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:55.538770 25545 raft_consensus.cc:348] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:55.539016 25545 raft_consensus.cc:374] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:55.539115 25545 raft_consensus.cc:729] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Initialized, Role: FOLLOWER
I20240503 08:11:55.539408 25545 consensus_queue.cc:260] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:55.540097 25545 ts_tablet_manager.cc:1398] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:55.540343 25544 raft_consensus.cc:348] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:55.540617 25544 raft_consensus.cc:374] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:55.540774 25544 raft_consensus.cc:729] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Initialized, Role: FOLLOWER
I20240503 08:11:55.541090 25544 consensus_queue.cc:260] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:55.541893 25544 ts_tablet_manager.cc:1398] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:11:55.542198 25544 tablet_bootstrap.cc:492] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42: Bootstrap starting.
I20240503 08:11:55.544137 25544 tablet_bootstrap.cc:654] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:55.545415 25543 tablet_bootstrap.cc:492] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6: No bootstrap required, opened a new log
I20240503 08:11:55.545652 25543 ts_tablet_manager.cc:1367] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 08:11:55.546494 25543 raft_consensus.cc:348] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:55.546667 25543 raft_consensus.cc:374] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:55.546751 25543 raft_consensus.cc:729] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Initialized, Role: FOLLOWER
I20240503 08:11:55.546983 25543 consensus_queue.cc:260] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:55.547575 25543 ts_tablet_manager.cc:1398] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:55.547919 25543 tablet_bootstrap.cc:492] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6: Bootstrap starting.
I20240503 08:11:55.549649 25543 tablet_bootstrap.cc:654] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:55.552146 25544 tablet_bootstrap.cc:492] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42: No bootstrap required, opened a new log
I20240503 08:11:55.552325 25544 ts_tablet_manager.cc:1367] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240503 08:11:55.553081 25544 raft_consensus.cc:348] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:55.553298 25544 raft_consensus.cc:374] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:55.553375 25544 raft_consensus.cc:729] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Initialized, Role: FOLLOWER
I20240503 08:11:55.553611 25544 consensus_queue.cc:260] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:55.554345 25544 ts_tablet_manager.cc:1398] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:55.556236 25543 tablet_bootstrap.cc:492] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6: No bootstrap required, opened a new log
I20240503 08:11:55.556393 25543 ts_tablet_manager.cc:1367] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240503 08:11:55.557284 25543 raft_consensus.cc:348] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:55.557438 25543 raft_consensus.cc:374] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:55.557528 25543 raft_consensus.cc:729] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Initialized, Role: FOLLOWER
I20240503 08:11:55.557749 25543 consensus_queue.cc:260] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:55.558216 25543 ts_tablet_manager.cc:1398] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:11:55.631803 25430 raft_consensus.cc:482] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:55.632019 25430 raft_consensus.cc:504] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:55.632685 25430 leader_election.cc:288] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:11:55.638397 24647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83697ce9030f4981af717d18eb6757a4" candidate_uuid: "b53502e2a1ae4c86a054d38048dd0df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" is_pre_election: true
I20240503 08:11:55.638629 24647 raft_consensus.cc:2454] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b53502e2a1ae4c86a054d38048dd0df6 in term 0.
I20240503 08:11:55.638975 24320 leader_election.cc:302] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:55.639246 25430 raft_consensus.cc:2781] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:55.639360 25430 raft_consensus.cc:482] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:55.639451 25430 raft_consensus.cc:3037] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:55.641757 25430 raft_consensus.cc:504] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:55.642225 25430 leader_election.cc:288] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 1 election: Requested vote from peers 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:11:55.642669 24647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83697ce9030f4981af717d18eb6757a4" candidate_uuid: "b53502e2a1ae4c86a054d38048dd0df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42"
I20240503 08:11:55.642865 24647 raft_consensus.cc:3037] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:55.644699 24647 raft_consensus.cc:2454] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b53502e2a1ae4c86a054d38048dd0df6 in term 1.
I20240503 08:11:55.645073 24320 leader_election.cc:302] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:55.645339 25430 raft_consensus.cc:2781] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:55.645709 25430 raft_consensus.cc:686] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 LEADER]: Becoming Leader. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Running, Role: LEADER
I20240503 08:11:55.645948 25430 consensus_queue.cc:237] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:55.648260 24242 catalog_manager.cc:5484] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129). New cstate: current_term: 1 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:55.661095 25471 raft_consensus.cc:482] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:55.661300 25471 raft_consensus.cc:504] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:55.661880 25471 leader_election.cc:288] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:55.662451 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73b0d5398cd484ba45515cf95cd5b14" candidate_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6" is_pre_election: true
I20240503 08:11:55.662667 24386 raft_consensus.cc:2454] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56fb8e40a4da4b3794f07ad8137ecf42 in term 0.
I20240503 08:11:55.663048 24580 leader_election.cc:302] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:55.663327 25471 raft_consensus.cc:2781] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:55.663430 25471 raft_consensus.cc:482] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:55.663517 25471 raft_consensus.cc:3037] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:55.665439 25471 raft_consensus.cc:504] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:55.665866 25471 leader_election.cc:288] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 election: Requested vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:55.666126 25530 raft_consensus.cc:482] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:55.666321 25530 raft_consensus.cc:504] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:55.666399 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73b0d5398cd484ba45515cf95cd5b14" candidate_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
I20240503 08:11:55.666595 24386 raft_consensus.cc:3037] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:55.666913 25530 leader_election.cc:288] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:55.667452 24385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a32af7adaf410582e1b898257db62d" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6" is_pre_election: true
I20240503 08:11:55.667675 24385 raft_consensus.cc:2454] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a640901590545eeb69e216d2399ea51 in term 0.
I20240503 08:11:55.668090 24450 leader_election.cc:302] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:55.668372 25530 raft_consensus.cc:2781] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:55.668480 25530 raft_consensus.cc:482] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:55.668561 25530 raft_consensus.cc:3037] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:55.669257 24386 raft_consensus.cc:2454] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56fb8e40a4da4b3794f07ad8137ecf42 in term 1.
I20240503 08:11:55.669703 24580 leader_election.cc:302] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:55.669991 25471 raft_consensus.cc:2781] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:55.670135 25471 raft_consensus.cc:686] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Becoming Leader. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Running, Role: LEADER
I20240503 08:11:55.670692 25471 consensus_queue.cc:237] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } }
I20240503 08:11:55.671007 25530 raft_consensus.cc:504] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:55.671563 25530 leader_election.cc:288] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 election: Requested vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:11:55.672188 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a32af7adaf410582e1b898257db62d" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
I20240503 08:11:55.672431 24386 raft_consensus.cc:3037] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:55.673528 24242 catalog_manager.cc:5484] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131). New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:55.674998 24386 raft_consensus.cc:2454] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a640901590545eeb69e216d2399ea51 in term 1.
I20240503 08:11:55.675412 24450 leader_election.cc:302] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:11:55.675738 25530 raft_consensus.cc:2781] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:55.675873 25530 raft_consensus.cc:686] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Becoming Leader. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Running, Role: LEADER
I20240503 08:11:55.676110 25530 consensus_queue.cc:237] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:55.678803 24242 catalog_manager.cc:5484] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a640901590545eeb69e216d2399ea51 (127.20.107.130). New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } }
W20240503 08:11:56.053927 25558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:56.054260 25558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:56.062074 25558 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 08:11:56.130997 25530 consensus_queue.cc:1035] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:56.186024 25430 consensus_queue.cc:1035] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:56.287016 25471 consensus_queue.cc:1035] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 299.895 ms
time per row: 0.149947 ms
I20240503 08:11:56.522907 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.132:36475
--local_ip_for_outbound_sockets=127.20.107.132
--tserver_master_addrs=127.20.107.190:44243
--webserver_port=33449
--webserver_interface=127.20.107.132
--builtin_ntp_servers=127.20.107.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:56.654251 25529 raft_consensus.cc:482] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:56.654475 25529 raft_consensus.cc:504] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:56.655128 25529 leader_election.cc:288] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d221e07029452cae34b8920884507e (127.20.107.135:45329)
W20240503 08:11:56.656039 24452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45329: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240503 08:11:56.661218 24452 leader_election.cc:334] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a0d221e07029452cae34b8920884507e (127.20.107.135:45329): Network error: Client connection negotiation failed: client connection to 127.20.107.135:45329: connect: Connection refused (error 111)
I20240503 08:11:56.661414 24452 leader_election.cc:302] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6a640901590545eeb69e216d2399ea51; no voters: a0d221e07029452cae34b8920884507e
I20240503 08:11:56.661720 25529 raft_consensus.cc:2726] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:11:56.866101 25615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:56.866396 25615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:56.866484 25615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:56.874414 25615 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:56.874549 25615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:56.874672 25615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.132
I20240503 08:11:56.884387 25615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.132:36475
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.107.132
--webserver_port=33449
--tserver_master_addrs=127.20.107.190:44243
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.132
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:56.885802 25615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:56.887501 25615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:56.900519 25623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:56.901197 25625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:56.901768 25622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:56.902498 25615 server_base.cc:1034] running on GCE node
I20240503 08:11:56.903129 25615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:56.904093 25615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:56.905292 25615 hybrid_clock.cc:648] HybridClock initialized: now 1714723916905229 us; error 58 us; skew 500 ppm
I20240503 08:11:56.905584 25615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:56.907927 25615 webserver.cc:460] Webserver started at http://127.20.107.132:33449/ using document root <none> and password file <none>
I20240503 08:11:56.908651 25615 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:56.908787 25615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:56.914178 25615 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:56.917352 25631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:56.918740 25615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240503 08:11:56.918912 25615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e7416e7c03a74a24b8fe2508111770d7"
format_stamp: "Formatted at 2024-05-03 08:11:44 on dist-test-slave-4tl1"
I20240503 08:11:56.919831 25615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:56.937307 25615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:56.938148 25615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:56.938443 25615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:56.939388 25615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:56.941067 25615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:56.941169 25615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:56.941342 25615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:56.941406 25615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:56.986380 25615 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.132:36475
I20240503 08:11:56.986421 25743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.132:36475 every 8 connection(s)
I20240503 08:11:56.987773 25615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:56.998044 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 25615
I20240503 08:11:56.998770 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.133:45569
--local_ip_for_outbound_sockets=127.20.107.133
--tserver_master_addrs=127.20.107.190:44243
--webserver_port=45567
--webserver_interface=127.20.107.133
--builtin_ntp_servers=127.20.107.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:57.000351 25744 heartbeater.cc:347] Connected to a master server at 127.20.107.190:44243
I20240503 08:11:57.000624 25744 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:57.001466 25744 heartbeater.cc:511] Master 127.20.107.190:44243 requested a full tablet report, sending...
I20240503 08:11:57.003015 24240 ts_manager.cc:194] Re-registered known tserver with Master: e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132:36475)
I20240503 08:11:57.003945 24240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.132:53835
W20240503 08:11:57.341431 25748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:57.341743 25748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:57.341881 25748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:57.349377 25748 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:57.349519 25748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:57.349678 25748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.133
I20240503 08:11:57.359108 25748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.133:45569
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.107.133
--webserver_port=45567
--tserver_master_addrs=127.20.107.190:44243
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.133
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:57.360445 25748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:57.362417 25748 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:57.375746 25755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:57.376191 25757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:57.376883 25754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:57.377691 25748 server_base.cc:1034] running on GCE node
I20240503 08:11:57.378312 25748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:57.379294 25748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:57.380507 25748 hybrid_clock.cc:648] HybridClock initialized: now 1714723917380455 us; error 43 us; skew 500 ppm
I20240503 08:11:57.380849 25748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:57.383193 25748 webserver.cc:460] Webserver started at http://127.20.107.133:45567/ using document root <none> and password file <none>
I20240503 08:11:57.383956 25748 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:57.384074 25748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:57.389569 25748 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 08:11:57.392392 25763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:57.393810 25748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:57.393990 25748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f9705fdde3854639ba85097a161f37e7"
format_stamp: "Formatted at 2024-05-03 08:11:44 on dist-test-slave-4tl1"
I20240503 08:11:57.394841 25748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:57.432016 25748 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:57.432847 25748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:57.433210 25748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:57.434162 25748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:57.435627 25748 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:57.435698 25748 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:57.435787 25748 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:57.435829 25748 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:57.468323 25748 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.133:45569
I20240503 08:11:57.468369 25875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.133:45569 every 8 connection(s)
I20240503 08:11:57.469851 25748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:57.474454 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 25748
I20240503 08:11:57.475171 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.134:46393
--local_ip_for_outbound_sockets=127.20.107.134
--tserver_master_addrs=127.20.107.190:44243
--webserver_port=37899
--webserver_interface=127.20.107.134
--builtin_ntp_servers=127.20.107.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:57.483708 25876 heartbeater.cc:347] Connected to a master server at 127.20.107.190:44243
I20240503 08:11:57.484023 25876 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:57.484901 25876 heartbeater.cc:511] Master 127.20.107.190:44243 requested a full tablet report, sending...
I20240503 08:11:57.486609 24240 ts_manager.cc:194] Re-registered known tserver with Master: f9705fdde3854639ba85097a161f37e7 (127.20.107.133:45569)
I20240503 08:11:57.487654 24240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.133:51665
W20240503 08:11:57.824772 25879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:57.825134 25879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:57.825264 25879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:57.832971 25879 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:57.833120 25879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:57.833271 25879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.134
I20240503 08:11:57.842425 25879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.134:46393
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.107.134
--webserver_port=37899
--tserver_master_addrs=127.20.107.190:44243
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.134
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:57.843786 25879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:57.845587 25879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:57.858276 25885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:57.858786 25887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:57.859464 25884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:57.860234 25879 server_base.cc:1034] running on GCE node
I20240503 08:11:57.860915 25879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:57.861918 25879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:57.863214 25879 hybrid_clock.cc:648] HybridClock initialized: now 1714723917863130 us; error 61 us; skew 500 ppm
I20240503 08:11:57.863556 25879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:57.866243 25879 webserver.cc:460] Webserver started at http://127.20.107.134:37899/ using document root <none> and password file <none>
I20240503 08:11:57.867017 25879 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:57.867138 25879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:57.872800 25879 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:57.875766 25893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:57.877135 25879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:57.877303 25879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9d313001b0a647caba8f53540a2b67ac"
format_stamp: "Formatted at 2024-05-03 08:11:45 on dist-test-slave-4tl1"
I20240503 08:11:57.878126 25879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:57.912268 25879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:57.913154 25879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:57.913435 25879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:57.914362 25879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:57.916150 25879 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:57.916256 25879 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:57.916393 25879 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:57.916446 25879 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:57.951568 25879 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.134:46393
I20240503 08:11:57.951622 26005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.134:46393 every 8 connection(s)
I20240503 08:11:57.953076 25879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:11:57.963385 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 25879
I20240503 08:11:57.964288 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.135:45329
--local_ip_for_outbound_sockets=127.20.107.135
--tserver_master_addrs=127.20.107.190:44243
--webserver_port=36541
--webserver_interface=127.20.107.135
--builtin_ntp_servers=127.20.107.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:57.967387 26006 heartbeater.cc:347] Connected to a master server at 127.20.107.190:44243
I20240503 08:11:57.967716 26006 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:57.968587 26006 heartbeater.cc:511] Master 127.20.107.190:44243 requested a full tablet report, sending...
I20240503 08:11:57.970077 24240 ts_manager.cc:194] Re-registered known tserver with Master: 9d313001b0a647caba8f53540a2b67ac (127.20.107.134:46393)
I20240503 08:11:57.970865 24240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.134:46819
I20240503 08:11:58.006625 25744 heartbeater.cc:503] Master 127.20.107.190:44243 was elected leader, sending a full tablet report...
W20240503 08:11:58.316546 26009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:58.316855 26009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:58.317008 26009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:58.325420 26009 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:58.325601 26009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:58.325731 26009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.135
I20240503 08:11:58.335286 26009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.135:45329
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.107.135
--webserver_port=36541
--tserver_master_addrs=127.20.107.190:44243
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.135
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:58.336674 26009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:58.338562 26009 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:58.351747 26015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:58.352372 26014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:58.353003 26017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:58.353694 26009 server_base.cc:1034] running on GCE node
I20240503 08:11:58.354285 26009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:58.355296 26009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:58.356518 26009 hybrid_clock.cc:648] HybridClock initialized: now 1714723918356458 us; error 46 us; skew 500 ppm
I20240503 08:11:58.356842 26009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:58.359258 26009 webserver.cc:460] Webserver started at http://127.20.107.135:36541/ using document root <none> and password file <none>
I20240503 08:11:58.360080 26009 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:58.360219 26009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:58.366179 26009 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 08:11:58.369570 26023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:58.371249 26009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240503 08:11:58.371454 26009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a0d221e07029452cae34b8920884507e"
format_stamp: "Formatted at 2024-05-03 08:11:45 on dist-test-slave-4tl1"
I20240503 08:11:58.372372 26009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:58.388718 26009 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:58.389580 26009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:58.389880 26009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:58.390906 26009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:58.393249 26030 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 08:11:58.398629 26009 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:11:58.398772 26009 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20240503 08:11:58.398921 26009 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:11:58.401892 26009 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:11:58.401993 26009 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.001s sys 0.002s
I20240503 08:11:58.402367 26030 tablet_bootstrap.cc:492] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: Bootstrap starting.
I20240503 08:11:58.422271 26030 log.cc:824] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:58.427963 26030 tablet_bootstrap.cc:492] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:58.428930 26030 tablet_bootstrap.cc:492] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: Bootstrap complete.
I20240503 08:11:58.430902 26030 ts_tablet_manager.cc:1367] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.012s
I20240503 08:11:58.441923 26009 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.135:45329
I20240503 08:11:58.441974 26137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.135:45329 every 8 connection(s)
I20240503 08:11:58.443766 26009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:11:58.445180 26030 raft_consensus.cc:348] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:58.447081 26030 raft_consensus.cc:729] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d221e07029452cae34b8920884507e, State: Initialized, Role: FOLLOWER
I20240503 08:11:58.447983 26030 consensus_queue.cc:260] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:58.449995 26030 ts_tablet_manager.cc:1398] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
I20240503 08:11:58.451786 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 26009
I20240503 08:11:58.452080 20910 test_util.cc:268] Using random seed: 1175196993
I20240503 08:11:58.457464 26138 heartbeater.cc:347] Connected to a master server at 127.20.107.190:44243
I20240503 08:11:58.457768 26138 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:58.458699 26138 heartbeater.cc:511] Master 127.20.107.190:44243 requested a full tablet report, sending...
I20240503 08:11:58.461429 24240 ts_manager.cc:194] Re-registered known tserver with Master: a0d221e07029452cae34b8920884507e (127.20.107.135:45329)
I20240503 08:11:58.462929 24240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.135:37663
I20240503 08:11:58.466214 26138 heartbeater.cc:503] Master 127.20.107.190:44243 was elected leader, sending a full tablet report...
I20240503 08:11:58.472456 20910 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 08:11:58.473114 20910 test_util.cc:268] Using random seed: 1175218028
I20240503 08:11:58.490957 25876 heartbeater.cc:503] Master 127.20.107.190:44243 was elected leader, sending a full tablet report...
I20240503 08:11:58.497762 20910 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 08:11:58.500627 20910 test_util.cc:268] Using random seed: 1175245534
I20240503 08:11:58.503118 25529 raft_consensus.cc:482] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:58.503366 25529 raft_consensus.cc:504] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:11:58.504108 25529 leader_election.cc:288] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d221e07029452cae34b8920884507e (127.20.107.135:45329)
I20240503 08:11:58.540400 26093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d422797de9843edbc473fca1d8e9e87" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d221e07029452cae34b8920884507e" is_pre_election: true
I20240503 08:11:58.540975 26093 raft_consensus.cc:2396] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a640901590545eeb69e216d2399ea51 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 08:11:58.542218 24452 leader_election.cc:302] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6a640901590545eeb69e216d2399ea51; no voters: a0d221e07029452cae34b8920884507e
I20240503 08:11:58.542526 25529 raft_consensus.cc:2726] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:11:58.542676 20910 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 08:11:58.549280 20910 test_util.cc:268] Using random seed: 1175294124
I20240503 08:11:58.645738 20910 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 08:11:58.652730 20910 test_util.cc:268] Using random seed: 1175397627
I20240503 08:11:58.684688 20910 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240503 08:11:58.697747 26139 tablet.cc:2369] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:58.974447 26006 heartbeater.cc:503] Master 127.20.107.190:44243 was elected leader, sending a full tablet report...
I20240503 08:11:59.952683 24496 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:00.033710 24627 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:00.034451 26287 raft_consensus.cc:482] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:00.037056 26287 raft_consensus.cc:504] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:12:00.037302 25941 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:00.057737 26072 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:00.079195 26287 leader_election.cc:288] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:12:00.090452 25679 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:00.118626 24364 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:00.127557 24517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d422797de9843edbc473fca1d8e9e87" candidate_uuid: "a0d221e07029452cae34b8920884507e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a640901590545eeb69e216d2399ea51" is_pre_election: true
I20240503 08:12:00.127807 24517 raft_consensus.cc:2454] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d221e07029452cae34b8920884507e in term 1.
I20240503 08:12:00.134282 26025 leader_election.cc:302] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, a0d221e07029452cae34b8920884507e; no voters:
I20240503 08:12:00.136049 26287 raft_consensus.cc:2781] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:12:00.136219 26287 raft_consensus.cc:482] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:00.136337 26287 raft_consensus.cc:3037] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:00.142303 26287 raft_consensus.cc:504] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:12:00.143556 26287 leader_election.cc:288] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [CANDIDATE]: Term 2 election: Requested vote from peers 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:12:00.144639 24517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d422797de9843edbc473fca1d8e9e87" candidate_uuid: "a0d221e07029452cae34b8920884507e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a640901590545eeb69e216d2399ea51"
I20240503 08:12:00.144838 24517 raft_consensus.cc:3037] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:00.147522 24517 raft_consensus.cc:2454] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d221e07029452cae34b8920884507e in term 2.
I20240503 08:12:00.147959 26025 leader_election.cc:302] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, a0d221e07029452cae34b8920884507e; no voters:
I20240503 08:12:00.148375 26287 raft_consensus.cc:2781] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:00.165114 25811 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:00.168824 26287 raft_consensus.cc:686] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 2 LEADER]: Becoming Leader. State: Replica: a0d221e07029452cae34b8920884507e, State: Running, Role: LEADER
I20240503 08:12:00.169432 26287 consensus_queue.cc:237] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } }
I20240503 08:12:00.200201 24240 catalog_manager.cc:5484] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "a0d221e07029452cae34b8920884507e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:00.660460 24515 raft_consensus.cc:1261] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Refusing update from remote peer a0d221e07029452cae34b8920884507e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:12:00.661492 26287 consensus_queue.cc:1035] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:01.163897 24627 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:01.280754 24496 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:01.289871 25941 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:01.377434 24364 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:01.439389 26072 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:01.483662 25679 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:01.561686 25811 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:01.746969 24515 consensus_queue.cc:237] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } }
I20240503 08:12:01.754851 24647 raft_consensus.cc:1261] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
W20240503 08:12:01.763801 24450 consensus_peers.cc:485] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 -> Peer f9705fdde3854639ba85097a161f37e7 (127.20.107.133:45569): Couldn't send request to peer f9705fdde3854639ba85097a161f37e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0756a24eb5d4e26b2d44e46638a74be. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:01.765908 26345 consensus_queue.cc:1035] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240503 08:12:01.773687 26345 raft_consensus.cc:2932] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER f9705fdde3854639ba85097a161f37e7 (127.20.107.133) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } } }
I20240503 08:12:01.778789 24645 raft_consensus.cc:2932] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER f9705fdde3854639ba85097a161f37e7 (127.20.107.133) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } } }
I20240503 08:12:01.792717 24240 catalog_manager.cc:5484] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: config changed from index -1 to 66, NON_VOTER f9705fdde3854639ba85097a161f37e7 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } } }
I20240503 08:12:01.911474 24647 consensus_queue.cc:237] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 1.61, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } }
I20240503 08:12:01.918305 24382 raft_consensus.cc:1261] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 08:12:01.919006 26329 consensus_queue.cc:1035] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20240503 08:12:01.922448 25609 raft_consensus.cc:2932] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } } }
I20240503 08:12:01.929036 24244 catalog_manager.cc:5484] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: config changed from index -1 to 62, NON_VOTER e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) added. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:01.929322 24382 raft_consensus.cc:2932] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } } }
W20240503 08:12:01.937197 24581 consensus_peers.cc:485] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 -> Peer e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132:36475): Couldn't send request to peer e7416e7c03a74a24b8fe2508111770d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d2772d08c06498db2cb0b02497ea042. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:02.020022 24645 consensus_queue.cc:237] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } }
I20240503 08:12:02.037201 24382 raft_consensus.cc:1261] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 08:12:02.037879 25609 consensus_queue.cc:1035] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
W20240503 08:12:02.038838 24581 consensus_peers.cc:485] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 -> Peer 9d313001b0a647caba8f53540a2b67ac (127.20.107.134:46393): Couldn't send request to peer 9d313001b0a647caba8f53540a2b67ac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08df2ee2ca344cd2abd5ecb67ba9ffc1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:02.040282 26174 raft_consensus.cc:2932] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } } }
I20240503 08:12:02.046234 24385 raft_consensus.cc:2932] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } } }
I20240503 08:12:02.046663 24240 catalog_manager.cc:5484] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: config changed from index -1 to 79, NON_VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:02.090581 24645 consensus_queue.cc:237] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.55, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } }
I20240503 08:12:02.097653 24382 raft_consensus.cc:1261] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 08:12:02.098275 26174 consensus_queue.cc:1035] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20240503 08:12:02.101991 26252 raft_consensus.cc:2932] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.56: config changed from index -1 to 56, NON_VOTER a0d221e07029452cae34b8920884507e (127.20.107.135) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } } }
I20240503 08:12:02.103088 24385 raft_consensus.cc:2932] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index -1 to 56, NON_VOTER a0d221e07029452cae34b8920884507e (127.20.107.135) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } } }
I20240503 08:12:02.113725 24240 catalog_manager.cc:5484] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: config changed from index -1 to 56, NON_VOTER a0d221e07029452cae34b8920884507e (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:02.134913 24582 consensus_peers.cc:485] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 -> Peer a0d221e07029452cae34b8920884507e (127.20.107.135:45329): Couldn't send request to peer a0d221e07029452cae34b8920884507e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d73b0d5398cd484ba45515cf95cd5b14. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:02.213336 24645 consensus_queue.cc:237] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } }
W20240503 08:12:02.216024 24581 consensus_peers.cc:485] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 -> Peer 9d313001b0a647caba8f53540a2b67ac (127.20.107.134:46393): Couldn't send request to peer 9d313001b0a647caba8f53540a2b67ac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b858bcd6e1c469d9f19f142106908d3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:02.217097 24515 raft_consensus.cc:1261] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240503 08:12:02.217702 26329 consensus_queue.cc:1035] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240503 08:12:02.222807 26329 raft_consensus.cc:2932] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index -1 to 70, NON_VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } } }
I20240503 08:12:02.223534 24515 raft_consensus.cc:2932] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index -1 to 70, NON_VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } } }
I20240503 08:12:02.229557 24244 catalog_manager.cc:5484] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51 reported cstate change: config changed from index -1 to 70, NON_VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } } }
I20240503 08:12:02.254137 26387 ts_tablet_manager.cc:902] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7: Initiating tablet copy from peer 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:12:02.281816 26387 tablet_copy_client.cc:308] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:43793
I20240503 08:12:02.356770 24382 consensus_queue.cc:237] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.61, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } }
I20240503 08:12:02.369361 24537 tablet_copy_service.cc:140] P 6a640901590545eeb69e216d2399ea51: Received BeginTabletCopySession request for tablet e0756a24eb5d4e26b2d44e46638a74be from peer f9705fdde3854639ba85097a161f37e7 ({username='slave'} at 127.20.107.133:46723)
I20240503 08:12:02.369735 24537 tablet_copy_service.cc:161] P 6a640901590545eeb69e216d2399ea51: Beginning new tablet copy session on tablet e0756a24eb5d4e26b2d44e46638a74be from peer f9705fdde3854639ba85097a161f37e7 at {username='slave'} at 127.20.107.133:46723: session id = f9705fdde3854639ba85097a161f37e7-e0756a24eb5d4e26b2d44e46638a74be
I20240503 08:12:02.377743 24646 raft_consensus.cc:1261] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 08:12:02.377935 24537 tablet_copy_source_session.cc:205] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:02.380159 26327 consensus_queue.cc:1035] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 60, Time since last communication: 0.001s
I20240503 08:12:02.385576 26327 raft_consensus.cc:2932] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 LEADER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER f9705fdde3854639ba85097a161f37e7 (127.20.107.133) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } } }
I20240503 08:12:02.392416 24645 raft_consensus.cc:2932] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER f9705fdde3854639ba85097a161f37e7 (127.20.107.133) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } } }
I20240503 08:12:02.393961 24244 catalog_manager.cc:5484] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: config changed from index -1 to 62, NON_VOTER f9705fdde3854639ba85097a161f37e7 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:02.394474 26387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0756a24eb5d4e26b2d44e46638a74be. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:12:02.409150 24319 consensus_peers.cc:485] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 -> Peer f9705fdde3854639ba85097a161f37e7 (127.20.107.133:45569): Couldn't send request to peer f9705fdde3854639ba85097a161f37e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83697ce9030f4981af717d18eb6757a4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:02.448856 26412 ts_tablet_manager.cc:902] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7: Initiating tablet copy from peer 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:12:02.461053 26387 tablet_copy_client.cc:783] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:02.461964 26387 tablet_copy_client.cc:647] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:02.470911 26387 tablet_copy_client.cc:523] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:02.480633 26387 tablet_bootstrap.cc:492] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7: Bootstrap starting.
I20240503 08:12:02.542091 26417 ts_tablet_manager.cc:902] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e: Initiating tablet copy from peer 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:12:02.547881 26387 log.cc:824] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:02.555516 26420 ts_tablet_manager.cc:902] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac: Initiating tablet copy from peer 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:12:02.603964 26412 tablet_copy_client.cc:308] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:34003
I20240503 08:12:02.605167 26417 tablet_copy_client.cc:308] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:34003
I20240503 08:12:02.626804 24515 consensus_queue.cc:237] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 1.67, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } }
I20240503 08:12:02.637853 24667 tablet_copy_service.cc:140] P 56fb8e40a4da4b3794f07ad8137ecf42: Received BeginTabletCopySession request for tablet 7d2772d08c06498db2cb0b02497ea042 from peer e7416e7c03a74a24b8fe2508111770d7 ({username='slave'} at 127.20.107.132:41209)
I20240503 08:12:02.638167 24667 tablet_copy_service.cc:161] P 56fb8e40a4da4b3794f07ad8137ecf42: Beginning new tablet copy session on tablet 7d2772d08c06498db2cb0b02497ea042 from peer e7416e7c03a74a24b8fe2508111770d7 at {username='slave'} at 127.20.107.132:41209: session id = e7416e7c03a74a24b8fe2508111770d7-7d2772d08c06498db2cb0b02497ea042
I20240503 08:12:02.647305 24667 tablet_copy_source_session.cc:205] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:02.649271 24666 tablet_copy_service.cc:140] P 56fb8e40a4da4b3794f07ad8137ecf42: Received BeginTabletCopySession request for tablet d73b0d5398cd484ba45515cf95cd5b14 from peer a0d221e07029452cae34b8920884507e ({username='slave'} at 127.20.107.135:33925)
I20240503 08:12:02.649518 24666 tablet_copy_service.cc:161] P 56fb8e40a4da4b3794f07ad8137ecf42: Beginning new tablet copy session on tablet d73b0d5398cd484ba45515cf95cd5b14 from peer a0d221e07029452cae34b8920884507e at {username='slave'} at 127.20.107.135:33925: session id = a0d221e07029452cae34b8920884507e-d73b0d5398cd484ba45515cf95cd5b14
I20240503 08:12:02.654515 24666 tablet_copy_source_session.cc:205] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:02.661731 26417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73b0d5398cd484ba45515cf95cd5b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:02.669237 26412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2772d08c06498db2cb0b02497ea042. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:02.687732 24645 raft_consensus.cc:1261] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch)
I20240503 08:12:02.691351 25529 consensus_queue.cc:1035] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20240503 08:12:02.695308 26420 tablet_copy_client.cc:308] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:34003
I20240503 08:12:02.696250 26182 raft_consensus.cc:2932] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.68: config changed from index -1 to 68, NON_VOTER e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) added. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } } }
I20240503 08:12:02.703954 24645 raft_consensus.cc:2932] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index -1 to 68, NON_VOTER e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) added. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } } }
I20240503 08:12:02.706220 24244 catalog_manager.cc:5484] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 reported cstate change: config changed from index -1 to 68, NON_VOTER e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) added. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:02.714429 26417 tablet_copy_client.cc:783] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:02.722380 24666 tablet_copy_service.cc:140] P 56fb8e40a4da4b3794f07ad8137ecf42: Received BeginTabletCopySession request for tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1 from peer 9d313001b0a647caba8f53540a2b67ac ({username='slave'} at 127.20.107.134:35505)
I20240503 08:12:02.722564 24666 tablet_copy_service.cc:161] P 56fb8e40a4da4b3794f07ad8137ecf42: Beginning new tablet copy session on tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1 from peer 9d313001b0a647caba8f53540a2b67ac at {username='slave'} at 127.20.107.134:35505: session id = 9d313001b0a647caba8f53540a2b67ac-08df2ee2ca344cd2abd5ecb67ba9ffc1
I20240503 08:12:02.725890 24666 tablet_copy_source_session.cc:205] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:02.715359 26417 tablet_copy_client.cc:647] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:02.730167 26417 tablet_copy_client.cc:523] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:02.731863 26420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:02.745752 26412 tablet_copy_client.cc:783] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:02.746469 26412 tablet_copy_client.cc:647] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:02.752010 26412 tablet_copy_client.cc:523] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:02.757136 26412 tablet_bootstrap.cc:492] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7: Bootstrap starting.
I20240503 08:12:02.761236 26417 tablet_bootstrap.cc:492] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e: Bootstrap starting.
W20240503 08:12:02.770331 24450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.129:46775, user_credentials={real_user=slave}} blocked reactor thread for 67773.2us
I20240503 08:12:02.788993 26420 tablet_copy_client.cc:783] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:02.789609 26420 tablet_copy_client.cc:647] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:02.792207 26420 tablet_copy_client.cc:523] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:02.801244 26420 tablet_bootstrap.cc:492] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac: Bootstrap starting.
I20240503 08:12:02.807247 26439 ts_tablet_manager.cc:902] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac: Initiating tablet copy from peer 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:12:02.819612 26412 log.cc:824] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7: Log is configured to *not* fsync() on all Append() calls
W20240503 08:12:02.838279 24451 consensus_peers.cc:485] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 -> Peer e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132:36475): Couldn't send request to peer e7416e7c03a74a24b8fe2508111770d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3071edf0c1004a6496d1cb46591d422e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:02.889592 24516 consensus_queue.cc:237] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 1.88, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } }
W20240503 08:12:02.892048 24452 consensus_peers.cc:485] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 -> Peer a0d221e07029452cae34b8920884507e (127.20.107.135:45329): Couldn't send request to peer a0d221e07029452cae34b8920884507e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61bf5a9f26034821b821a45ac37ba133. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:02.897841 24645 raft_consensus.cc:1261] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch)
I20240503 08:12:02.900609 26345 consensus_queue.cc:1035] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20240503 08:12:02.902665 26420 log.cc:824] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:02.903956 26439 tablet_copy_client.cc:308] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:34003
I20240503 08:12:02.913426 25529 raft_consensus.cc:2932] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.89: config changed from index -1 to 89, NON_VOTER a0d221e07029452cae34b8920884507e (127.20.107.135) added. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } } }
I20240503 08:12:02.920106 24240 catalog_manager.cc:5484] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 reported cstate change: config changed from index -1 to 89, NON_VOTER a0d221e07029452cae34b8920884507e (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:02.925377 24666 tablet_copy_service.cc:140] P 56fb8e40a4da4b3794f07ad8137ecf42: Received BeginTabletCopySession request for tablet 7b858bcd6e1c469d9f19f142106908d3 from peer 9d313001b0a647caba8f53540a2b67ac ({username='slave'} at 127.20.107.134:35505)
I20240503 08:12:02.925534 24666 tablet_copy_service.cc:161] P 56fb8e40a4da4b3794f07ad8137ecf42: Beginning new tablet copy session on tablet 7b858bcd6e1c469d9f19f142106908d3 from peer 9d313001b0a647caba8f53540a2b67ac at {username='slave'} at 127.20.107.134:35505: session id = 9d313001b0a647caba8f53540a2b67ac-7b858bcd6e1c469d9f19f142106908d3
I20240503 08:12:02.928298 24666 tablet_copy_source_session.cc:205] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:02.937433 26439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b858bcd6e1c469d9f19f142106908d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:02.943728 24646 raft_consensus.cc:2932] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index -1 to 89, NON_VOTER a0d221e07029452cae34b8920884507e (127.20.107.135) added. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } } }
I20240503 08:12:02.957497 26439 tablet_copy_client.cc:783] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:02.957844 26439 tablet_copy_client.cc:647] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:02.961040 26439 tablet_copy_client.cc:523] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:02.965006 26439 tablet_bootstrap.cc:492] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac: Bootstrap starting.
I20240503 08:12:02.986896 26387 tablet_bootstrap.cc:492] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=77 overwritten=0 applied=77 ignored=0} inserts{seen=708 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:02.988634 26387 tablet_bootstrap.cc:492] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7: Bootstrap complete.
I20240503 08:12:02.990211 26387 ts_tablet_manager.cc:1367] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7: Time spent bootstrapping tablet: real 0.510s user 0.161s sys 0.010s
I20240503 08:12:03.007776 26387 raft_consensus.cc:348] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } }
I20240503 08:12:03.009676 26387 raft_consensus.cc:729] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f9705fdde3854639ba85097a161f37e7, State: Initialized, Role: LEARNER
I20240503 08:12:03.013303 26387 consensus_queue.cc:260] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 77, Last appended: 1.77, Last appended by leader: 77, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } }
I20240503 08:12:03.028613 26387 ts_tablet_manager.cc:1398] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7: Time spent starting tablet: real 0.036s user 0.011s sys 0.000s
I20240503 08:12:03.033855 24537 tablet_copy_service.cc:342] P 6a640901590545eeb69e216d2399ea51: Request end of tablet copy session f9705fdde3854639ba85097a161f37e7-e0756a24eb5d4e26b2d44e46638a74be received from {username='slave'} at 127.20.107.133:46723
I20240503 08:12:03.034080 24537 tablet_copy_service.cc:433] P 6a640901590545eeb69e216d2399ea51: ending tablet copy session f9705fdde3854639ba85097a161f37e7-e0756a24eb5d4e26b2d44e46638a74be on tablet e0756a24eb5d4e26b2d44e46638a74be with peer f9705fdde3854639ba85097a161f37e7
I20240503 08:12:03.036800 26387 ts_tablet_manager.cc:902] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7: Initiating tablet copy from peer b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:12:03.066025 26457 ts_tablet_manager.cc:902] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7: Initiating tablet copy from peer 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:12:03.211524 24515 consensus_queue.cc:237] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } }
I20240503 08:12:03.214283 25831 raft_consensus.cc:1203] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.65->[1.66-1.81] Dedup: 1.77->[1.78-1.81]
I20240503 08:12:03.216640 26420 tablet_bootstrap.cc:492] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=88 overwritten=0 applied=88 ignored=0} inserts{seen=719 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:03.218600 26420 tablet_bootstrap.cc:492] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac: Bootstrap complete.
I20240503 08:12:03.219780 26420 ts_tablet_manager.cc:1367] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac: Time spent bootstrapping tablet: real 0.419s user 0.158s sys 0.015s
I20240503 08:12:03.223598 26387 tablet_copy_client.cc:308] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:46775
I20240503 08:12:03.236702 26420 raft_consensus.cc:348] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } }
I20240503 08:12:03.243311 26420 raft_consensus.cc:729] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d313001b0a647caba8f53540a2b67ac, State: Initialized, Role: LEARNER
I20240503 08:12:03.246191 26420 consensus_queue.cc:260] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 88, Last appended: 1.88, Last appended by leader: 88, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } }
I20240503 08:12:03.254590 24406 tablet_copy_service.cc:140] P b53502e2a1ae4c86a054d38048dd0df6: Received BeginTabletCopySession request for tablet 83697ce9030f4981af717d18eb6757a4 from peer f9705fdde3854639ba85097a161f37e7 ({username='slave'} at 127.20.107.133:55825)
I20240503 08:12:03.254914 24406 tablet_copy_service.cc:161] P b53502e2a1ae4c86a054d38048dd0df6: Beginning new tablet copy session on tablet 83697ce9030f4981af717d18eb6757a4 from peer f9705fdde3854639ba85097a161f37e7 at {username='slave'} at 127.20.107.133:55825: session id = f9705fdde3854639ba85097a161f37e7-83697ce9030f4981af717d18eb6757a4
I20240503 08:12:03.259378 26457 tablet_copy_client.cc:308] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:43793
I20240503 08:12:03.304545 24384 raft_consensus.cc:1261] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240503 08:12:03.305539 24406 tablet_copy_source_session.cc:205] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:03.308106 26387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83697ce9030f4981af717d18eb6757a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:03.314108 26387 tablet_copy_client.cc:783] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:03.314393 26387 tablet_copy_client.cc:647] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:03.317166 26387 tablet_copy_client.cc:523] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:03.319334 24537 tablet_copy_service.cc:140] P 6a640901590545eeb69e216d2399ea51: Received BeginTabletCopySession request for tablet 3071edf0c1004a6496d1cb46591d422e from peer e7416e7c03a74a24b8fe2508111770d7 ({username='slave'} at 127.20.107.132:44973)
I20240503 08:12:03.319517 24537 tablet_copy_service.cc:161] P 6a640901590545eeb69e216d2399ea51: Beginning new tablet copy session on tablet 3071edf0c1004a6496d1cb46591d422e from peer e7416e7c03a74a24b8fe2508111770d7 at {username='slave'} at 127.20.107.132:44973: session id = e7416e7c03a74a24b8fe2508111770d7-3071edf0c1004a6496d1cb46591d422e
I20240503 08:12:03.320726 26387 tablet_bootstrap.cc:492] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7: Bootstrap starting.
I20240503 08:12:03.322353 24537 tablet_copy_source_session.cc:205] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:03.322491 26420 ts_tablet_manager.cc:1398] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac: Time spent starting tablet: real 0.102s user 0.034s sys 0.025s
I20240503 08:12:03.323776 24666 tablet_copy_service.cc:342] P 56fb8e40a4da4b3794f07ad8137ecf42: Request end of tablet copy session 9d313001b0a647caba8f53540a2b67ac-08df2ee2ca344cd2abd5ecb67ba9ffc1 received from {username='slave'} at 127.20.107.134:35505
I20240503 08:12:03.323889 26457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3071edf0c1004a6496d1cb46591d422e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:03.323961 24666 tablet_copy_service.cc:433] P 56fb8e40a4da4b3794f07ad8137ecf42: ending tablet copy session 9d313001b0a647caba8f53540a2b67ac-08df2ee2ca344cd2abd5ecb67ba9ffc1 on tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1 with peer 9d313001b0a647caba8f53540a2b67ac
I20240503 08:12:03.330144 25529 consensus_queue.cc:1035] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
W20240503 08:12:03.331157 24451 consensus_peers.cc:485] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 -> Peer 9d313001b0a647caba8f53540a2b67ac (127.20.107.134:46393): Couldn't send request to peer 9d313001b0a647caba8f53540a2b67ac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8a32af7adaf410582e1b898257db62d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:03.333987 26457 tablet_copy_client.cc:783] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:03.334308 26457 tablet_copy_client.cc:647] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:03.337528 26457 tablet_copy_client.cc:523] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:03.351466 26345 raft_consensus.cc:1050] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51: attempting to promote NON_VOTER f9705fdde3854639ba85097a161f37e7 to VOTER
I20240503 08:12:03.352102 26457 tablet_bootstrap.cc:492] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7: Bootstrap starting.
I20240503 08:12:03.353057 26345 consensus_queue.cc:237] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } }
I20240503 08:12:03.356199 24645 raft_consensus.cc:1261] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240503 08:12:03.356946 26182 consensus_queue.cc:1035] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20240503 08:12:03.359445 26345 raft_consensus.cc:2932] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index 66 to 83, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:03.361194 24645 raft_consensus.cc:2932] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index 66 to 83, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:03.370891 24240 catalog_manager.cc:5484] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: config changed from index 66 to 83, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:03.372819 26237 raft_consensus.cc:2932] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index -1 to 71, NON_VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } } }
I20240503 08:12:03.375967 25831 raft_consensus.cc:1261] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [term 1 LEARNER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240503 08:12:03.375974 24385 raft_consensus.cc:2932] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index -1 to 71, NON_VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } } }
I20240503 08:12:03.388093 24243 catalog_manager.cc:5484] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 reported cstate change: config changed from index -1 to 71, NON_VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:03.434278 25961 raft_consensus.cc:1203] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Deduplicated request from leader. Original: 1.78->[1.79-1.99] Dedup: 1.88->[1.89-1.99]
I20240503 08:12:03.445468 26345 consensus_queue.cc:1035] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20240503 08:12:03.455207 25831 raft_consensus.cc:2932] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index 66 to 83, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:03.486526 26387 tablet_bootstrap.cc:492] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=76 overwritten=0 applied=76 ignored=0} inserts{seen=761 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:03.487111 26387 tablet_bootstrap.cc:492] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7: Bootstrap complete.
I20240503 08:12:03.487476 26387 ts_tablet_manager.cc:1367] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7: Time spent bootstrapping tablet: real 0.167s user 0.136s sys 0.014s
I20240503 08:12:03.488584 26387 raft_consensus.cc:348] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } }
I20240503 08:12:03.488953 26387 raft_consensus.cc:729] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f9705fdde3854639ba85097a161f37e7, State: Initialized, Role: LEARNER
I20240503 08:12:03.489229 26387 consensus_queue.cc:260] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 76, Last appended: 1.76, Last appended by leader: 76, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } }
I20240503 08:12:03.490018 26387 ts_tablet_manager.cc:1398] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:03.490828 24406 tablet_copy_service.cc:342] P b53502e2a1ae4c86a054d38048dd0df6: Request end of tablet copy session f9705fdde3854639ba85097a161f37e7-83697ce9030f4981af717d18eb6757a4 received from {username='slave'} at 127.20.107.133:55825
I20240503 08:12:03.491017 24406 tablet_copy_service.cc:433] P b53502e2a1ae4c86a054d38048dd0df6: ending tablet copy session f9705fdde3854639ba85097a161f37e7-83697ce9030f4981af717d18eb6757a4 on tablet 83697ce9030f4981af717d18eb6757a4 with peer f9705fdde3854639ba85097a161f37e7
I20240503 08:12:03.525547 26487 ts_tablet_manager.cc:902] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e: Initiating tablet copy from peer 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:12:03.561787 26487 tablet_copy_client.cc:308] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:43793
I20240503 08:12:03.562924 24537 tablet_copy_service.cc:140] P 6a640901590545eeb69e216d2399ea51: Received BeginTabletCopySession request for tablet 61bf5a9f26034821b821a45ac37ba133 from peer a0d221e07029452cae34b8920884507e ({username='slave'} at 127.20.107.135:39019)
I20240503 08:12:03.563068 24537 tablet_copy_service.cc:161] P 6a640901590545eeb69e216d2399ea51: Beginning new tablet copy session on tablet 61bf5a9f26034821b821a45ac37ba133 from peer a0d221e07029452cae34b8920884507e at {username='slave'} at 127.20.107.135:39019: session id = a0d221e07029452cae34b8920884507e-61bf5a9f26034821b821a45ac37ba133
I20240503 08:12:03.566057 24537 tablet_copy_source_session.cc:205] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:03.570025 24516 consensus_queue.cc:237] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.85, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } }
I20240503 08:12:03.571950 24384 raft_consensus.cc:1261] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240503 08:12:03.572551 26237 consensus_queue.cc:1035] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 84, Time since last communication: 0.000s
I20240503 08:12:03.573566 26487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61bf5a9f26034821b821a45ac37ba133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:03.578912 26237 raft_consensus.cc:2932] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.86: config changed from index -1 to 86, NON_VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } } }
I20240503 08:12:03.584900 24384 raft_consensus.cc:2932] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index -1 to 86, NON_VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } } }
I20240503 08:12:03.585939 24243 catalog_manager.cc:5484] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 reported cstate change: config changed from index -1 to 86, NON_VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:03.597473 24451 consensus_peers.cc:485] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 -> Peer 9d313001b0a647caba8f53540a2b67ac (127.20.107.134:46393): Couldn't send request to peer 9d313001b0a647caba8f53540a2b67ac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c65da71faf34351885b5033d3d68835. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:03.652427 26487 tablet_copy_client.cc:783] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:03.654023 26487 tablet_copy_client.cc:647] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:03.657595 26487 tablet_copy_client.cc:523] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:03.667516 24647 consensus_queue.cc:237] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 1.98, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } }
I20240503 08:12:03.673856 24384 raft_consensus.cc:1261] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 08:12:03.674634 26329 consensus_queue.cc:1035] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240503 08:12:03.676502 26487 tablet_bootstrap.cc:492] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e: Bootstrap starting.
I20240503 08:12:03.677184 26329 raft_consensus.cc:2932] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.99: config changed from index -1 to 99, NON_VOTER f9705fdde3854639ba85097a161f37e7 (127.20.107.133) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } } }
I20240503 08:12:03.681859 24384 raft_consensus.cc:2932] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index -1 to 99, NON_VOTER f9705fdde3854639ba85097a161f37e7 (127.20.107.133) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } } }
I20240503 08:12:03.683511 26457 tablet_bootstrap.cc:492] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=75 overwritten=0 applied=75 ignored=0} inserts{seen=760 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:03.684188 26457 tablet_bootstrap.cc:492] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7: Bootstrap complete.
I20240503 08:12:03.684818 26457 ts_tablet_manager.cc:1367] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7: Time spent bootstrapping tablet: real 0.333s user 0.127s sys 0.012s
I20240503 08:12:03.687840 24243 catalog_manager.cc:5484] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: config changed from index -1 to 99, NON_VOTER f9705fdde3854639ba85097a161f37e7 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } } }
I20240503 08:12:03.692370 26457 raft_consensus.cc:348] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } }
I20240503 08:12:03.693037 26457 raft_consensus.cc:729] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e7416e7c03a74a24b8fe2508111770d7, State: Initialized, Role: LEARNER
I20240503 08:12:03.693749 26457 consensus_queue.cc:260] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 1.75, Last appended by leader: 75, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } }
I20240503 08:12:03.705847 26457 ts_tablet_manager.cc:1398] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7: Time spent starting tablet: real 0.021s user 0.015s sys 0.001s
I20240503 08:12:03.707903 24537 tablet_copy_service.cc:342] P 6a640901590545eeb69e216d2399ea51: Request end of tablet copy session e7416e7c03a74a24b8fe2508111770d7-3071edf0c1004a6496d1cb46591d422e received from {username='slave'} at 127.20.107.132:44973
I20240503 08:12:03.708073 24537 tablet_copy_service.cc:433] P 6a640901590545eeb69e216d2399ea51: ending tablet copy session e7416e7c03a74a24b8fe2508111770d7-3071edf0c1004a6496d1cb46591d422e on tablet 3071edf0c1004a6496d1cb46591d422e with peer e7416e7c03a74a24b8fe2508111770d7
I20240503 08:12:03.708845 26420 ts_tablet_manager.cc:902] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac: Initiating tablet copy from peer 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:12:03.716092 26400 raft_consensus.cc:1050] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42: attempting to promote NON_VOTER 9d313001b0a647caba8f53540a2b67ac to VOTER
I20240503 08:12:03.717551 26400 consensus_queue.cc:237] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 1.101, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
I20240503 08:12:03.720701 24383 raft_consensus.cc:1261] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240503 08:12:03.721479 25960 raft_consensus.cc:1261] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240503 08:12:03.721832 26401 consensus_queue.cc:1035] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20240503 08:12:03.723383 26252 consensus_queue.cc:1035] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20240503 08:12:03.737360 26329 raft_consensus.cc:2932] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.102: config changed from index 79 to 102, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:03.743127 25960 raft_consensus.cc:2932] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 79 to 102, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:03.744392 24244 catalog_manager.cc:5484] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: config changed from index 79 to 102, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:03.749068 24385 raft_consensus.cc:2932] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 79 to 102, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
W20240503 08:12:03.785442 24580 consensus_peers.cc:485] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 -> Peer f9705fdde3854639ba85097a161f37e7 (127.20.107.133:45569): Couldn't send request to peer f9705fdde3854639ba85097a161f37e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c85cf905ebfa4e7b91a0af9ed2fa77da. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:03.797984 24386 raft_consensus.cc:1203] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.100->[1.101-1.101] Dedup: 1.101->[]
I20240503 08:12:03.802311 25831 raft_consensus.cc:1203] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.61->[1.62-1.79] Dedup: 1.76->[1.77-1.79]
I20240503 08:12:03.812891 26420 tablet_copy_client.cc:308] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:43793
I20240503 08:12:03.861889 24537 tablet_copy_service.cc:140] P 6a640901590545eeb69e216d2399ea51: Received BeginTabletCopySession request for tablet d8a32af7adaf410582e1b898257db62d from peer 9d313001b0a647caba8f53540a2b67ac ({username='slave'} at 127.20.107.134:56731)
I20240503 08:12:03.862092 24537 tablet_copy_service.cc:161] P 6a640901590545eeb69e216d2399ea51: Beginning new tablet copy session on tablet d8a32af7adaf410582e1b898257db62d from peer 9d313001b0a647caba8f53540a2b67ac at {username='slave'} at 127.20.107.134:56731: session id = 9d313001b0a647caba8f53540a2b67ac-d8a32af7adaf410582e1b898257db62d
I20240503 08:12:03.865435 24537 tablet_copy_source_session.cc:205] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:03.866719 26420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a32af7adaf410582e1b898257db62d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:03.875589 26420 tablet_copy_client.cc:783] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:03.876003 26420 tablet_copy_client.cc:647] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:03.880019 26420 tablet_copy_client.cc:523] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:03.889470 26420 tablet_bootstrap.cc:492] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac: Bootstrap starting.
I20240503 08:12:03.911098 26327 raft_consensus.cc:1050] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6: attempting to promote NON_VOTER f9705fdde3854639ba85097a161f37e7 to VOTER
I20240503 08:12:03.911880 26327 consensus_queue.cc:237] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } }
I20240503 08:12:03.921123 25830 raft_consensus.cc:1261] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7 [term 1 LEARNER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240503 08:12:03.937306 26526 consensus_queue.cc:1035] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 08:12:03.947428 24646 raft_consensus.cc:1261] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240503 08:12:03.948415 25430 consensus_queue.cc:1035] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 08:12:03.967104 25430 raft_consensus.cc:2932] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index 62 to 83, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:03.982472 24243 catalog_manager.cc:5484] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: config changed from index 62 to 83, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:03.989410 24646 raft_consensus.cc:2932] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index 62 to 83, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:03.996049 25699 raft_consensus.cc:1203] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.67->[1.68-1.83] Dedup: 1.75->[1.76-1.83]
I20240503 08:12:04.030510 24645 consensus_queue.cc:237] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } }
I20240503 08:12:04.034567 26345 raft_consensus.cc:1050] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51: attempting to promote NON_VOTER e7416e7c03a74a24b8fe2508111770d7 to VOTER
I20240503 08:12:04.035265 26345 consensus_queue.cc:237] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.84, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } }
I20240503 08:12:04.037669 25699 raft_consensus.cc:1261] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240503 08:12:04.038275 25529 consensus_queue.cc:1035] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 08:12:04.039722 24384 raft_consensus.cc:1261] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240503 08:12:04.041198 25609 consensus_queue.cc:1035] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20240503 08:12:04.043932 25609 raft_consensus.cc:2932] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index -1 to 88, NON_VOTER e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } } }
I20240503 08:12:04.047866 24386 raft_consensus.cc:2932] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index -1 to 88, NON_VOTER e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } } }
I20240503 08:12:04.050489 24645 raft_consensus.cc:1261] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240503 08:12:04.050638 24243 catalog_manager.cc:5484] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: config changed from index -1 to 88, NON_VOTER e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) added. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:04.051527 24581 consensus_peers.cc:485] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 -> Peer e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132:36475): Couldn't send request to peer e7416e7c03a74a24b8fe2508111770d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5dd5613d173f4b1eb0eacf2fec3a3ff0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:04.050546 25529 raft_consensus.cc:2932] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.85: config changed from index 68 to 85, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:04.055780 26237 consensus_queue.cc:1035] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 08:12:04.056974 25698 raft_consensus.cc:2932] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 68 to 85, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:04.066363 24645 raft_consensus.cc:2932] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 68 to 85, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:04.066955 24244 catalog_manager.cc:5484] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 reported cstate change: config changed from index 68 to 85, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:04.076246 24515 consensus_queue.cc:237] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.86, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } }
W20240503 08:12:04.076579 24319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.133:45569, user_credentials={real_user=slave}} blocked reactor thread for 154789us
I20240503 08:12:04.079470 25698 raft_consensus.cc:1261] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240503 08:12:04.084374 26345 consensus_queue.cc:1035] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 85, Time since last communication: 0.000s
I20240503 08:12:04.088629 25830 raft_consensus.cc:2932] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index 62 to 83, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:04.107846 24382 consensus_queue.cc:237] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 1.84, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } }
I20240503 08:12:04.111723 25831 raft_consensus.cc:1261] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7 [term 1 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240503 08:12:04.112372 26182 raft_consensus.cc:2932] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.87: config changed from index 85 to 87, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:04.113626 26327 consensus_queue.cc:1035] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 08:12:04.116394 26536 ts_tablet_manager.cc:902] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac: Initiating tablet copy from peer 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:12:04.117005 24230 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3071edf0c1004a6496d1cb46591d422e with cas_config_opid_index 85: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:04.120404 24240 catalog_manager.cc:5484] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 reported cstate change: config changed from index 85 to 87, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:04.122305 25699 raft_consensus.cc:2932] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 85 to 87, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:04.123695 26388 raft_consensus.cc:2932] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 LEADER]: Committing config change with OpId 1.85: config changed from index 83 to 85, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:04.124742 24627 tablet_service.cc:1514] Processing DeleteTablet for tablet 3071edf0c1004a6496d1cb46591d422e with delete_type TABLET_DATA_TOMBSTONED (TS 56fb8e40a4da4b3794f07ad8137ecf42 not found in new config with opid_index 87) from {username='slave'} at 127.0.0.1:46200
I20240503 08:12:04.132889 24229 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 83697ce9030f4981af717d18eb6757a4 with cas_config_opid_index 83: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:04.150846 25830 raft_consensus.cc:2932] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7 [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 83 to 85, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
W20240503 08:12:04.151897 26417 log.cc:924] Time spent T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e: Append to log took a long time: real 0.059s user 0.004s sys 0.000s
I20240503 08:12:04.166910 26539 tablet_replica.cc:330] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42: stopping tablet replica
I20240503 08:12:04.167582 26539 raft_consensus.cc:2229] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:04.168655 26539 raft_consensus.cc:2258] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:04.142212 24243 catalog_manager.cc:5484] T 83697ce9030f4981af717d18eb6757a4 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: config changed from index 83 to 85, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New cstate: current_term: 1 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:04.189956 24627 tablet_service.cc:1514] Processing DeleteTablet for tablet 83697ce9030f4981af717d18eb6757a4 with delete_type TABLET_DATA_TOMBSTONED (TS 56fb8e40a4da4b3794f07ad8137ecf42 not found in new config with opid_index 85) from {username='slave'} at 127.0.0.1:46200
I20240503 08:12:04.194082 26539 ts_tablet_manager.cc:1875] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:04.195621 26439 tablet_bootstrap.cc:492] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=78 overwritten=0 applied=78 ignored=0} inserts{seen=709 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:04.197983 26439 tablet_bootstrap.cc:492] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac: Bootstrap complete.
I20240503 08:12:04.199256 26439 ts_tablet_manager.cc:1367] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac: Time spent bootstrapping tablet: real 1.234s user 0.139s sys 0.012s
I20240503 08:12:04.202811 26439 raft_consensus.cc:348] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } }
I20240503 08:12:04.205348 26439 raft_consensus.cc:729] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d313001b0a647caba8f53540a2b67ac, State: Initialized, Role: LEARNER
I20240503 08:12:04.206243 26439 consensus_queue.cc:260] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 78, Last appended: 1.78, Last appended by leader: 78, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } }
I20240503 08:12:04.212167 26412 tablet_bootstrap.cc:492] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=73 overwritten=0 applied=73 ignored=0} inserts{seen=758 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:04.220172 26439 ts_tablet_manager.cc:1398] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240503 08:12:04.221598 24666 tablet_copy_service.cc:342] P 56fb8e40a4da4b3794f07ad8137ecf42: Request end of tablet copy session 9d313001b0a647caba8f53540a2b67ac-7b858bcd6e1c469d9f19f142106908d3 received from {username='slave'} at 127.20.107.134:35505
I20240503 08:12:04.221765 24666 tablet_copy_service.cc:433] P 56fb8e40a4da4b3794f07ad8137ecf42: ending tablet copy session 9d313001b0a647caba8f53540a2b67ac-7b858bcd6e1c469d9f19f142106908d3 on tablet 7b858bcd6e1c469d9f19f142106908d3 with peer 9d313001b0a647caba8f53540a2b67ac
I20240503 08:12:04.228353 26539 ts_tablet_manager.cc:1888] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.85
I20240503 08:12:04.229157 26412 tablet_bootstrap.cc:492] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7: Bootstrap complete.
I20240503 08:12:04.235924 26412 ts_tablet_manager.cc:1367] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7: Time spent bootstrapping tablet: real 1.479s user 0.166s sys 0.009s
I20240503 08:12:04.237907 26412 raft_consensus.cc:348] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } }
I20240503 08:12:04.241820 26412 raft_consensus.cc:729] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e7416e7c03a74a24b8fe2508111770d7, State: Initialized, Role: LEARNER
I20240503 08:12:04.243309 26412 consensus_queue.cc:260] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 73, Last appended: 1.73, Last appended by leader: 73, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } }
I20240503 08:12:04.250548 26412 ts_tablet_manager.cc:1398] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 08:12:04.251719 26093 consensus_queue.cc:237] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } }
I20240503 08:12:04.252029 24666 tablet_copy_service.cc:342] P 56fb8e40a4da4b3794f07ad8137ecf42: Request end of tablet copy session e7416e7c03a74a24b8fe2508111770d7-7d2772d08c06498db2cb0b02497ea042 received from {username='slave'} at 127.20.107.132:41209
I20240503 08:12:04.253888 24666 tablet_copy_service.cc:433] P 56fb8e40a4da4b3794f07ad8137ecf42: ending tablet copy session e7416e7c03a74a24b8fe2508111770d7-7d2772d08c06498db2cb0b02497ea042 on tablet 7d2772d08c06498db2cb0b02497ea042 with peer e7416e7c03a74a24b8fe2508111770d7
I20240503 08:12:04.254388 26539 log.cc:1196] T 3071edf0c1004a6496d1cb46591d422e P 56fb8e40a4da4b3794f07ad8137ecf42: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/3071edf0c1004a6496d1cb46591d422e
I20240503 08:12:04.256953 25960 raft_consensus.cc:1203] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Deduplicated request from leader. Original: 1.69->[1.70-1.87] Dedup: 1.78->[1.79-1.87]
I20240503 08:12:04.281700 26542 ts_tablet_manager.cc:902] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7: Initiating tablet copy from peer 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:12:04.294607 26536 tablet_copy_client.cc:308] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:43793
I20240503 08:12:04.296037 24537 tablet_copy_service.cc:140] P 6a640901590545eeb69e216d2399ea51: Received BeginTabletCopySession request for tablet 3c65da71faf34351885b5033d3d68835 from peer 9d313001b0a647caba8f53540a2b67ac ({username='slave'} at 127.20.107.134:56731)
I20240503 08:12:04.296224 24537 tablet_copy_service.cc:161] P 6a640901590545eeb69e216d2399ea51: Beginning new tablet copy session on tablet 3c65da71faf34351885b5033d3d68835 from peer 9d313001b0a647caba8f53540a2b67ac at {username='slave'} at 127.20.107.134:56731: session id = 9d313001b0a647caba8f53540a2b67ac-3c65da71faf34351885b5033d3d68835
I20240503 08:12:04.297174 26539 tablet_replica.cc:330] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42: stopping tablet replica
I20240503 08:12:04.297371 26539 raft_consensus.cc:2229] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:04.298022 26539 pending_rounds.cc:62] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42: Trying to abort 1 pending ops.
I20240503 08:12:04.298120 26539 pending_rounds.cc:68] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42: Aborting op as it isn't in flight: id { term: 1 index: 84 } timestamp: 7023509193072668672 op_type: WRITE_OP write_request { tablet_id: "83697ce9030f4981af717d18eb6757a4" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\310\036d\003\024{\334\233\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""6a6a4b5450ef1b1a" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023509192241483776 authz_token { token_data: "\010\372\272\322\261\006\"3\n\005slave\022*\n 72d93aae96774cf887b14b98c748dffc\020\001\030\001 \001(\001" signature: "<redacted>""s]\267\004d\245\2703\2440\346\023\263\267\016xY\327\270\304\037\'\311D\271u\'\372\306\336\367b\001R_r\300Wco\341\257\227\003\356\251\364<{sv\363\373\227\305\353/\375\232\014k\235?\314" signing_key_seq_num: 0 } } request_id { client_id: "9a9c7e884a454d8ab3339ea71aa1d244" seq_no: 235 first_incomplete_seq_no: 235 attempt_no: 0 }
I20240503 08:12:04.298748 26539 raft_consensus.cc:2258] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:04.299224 24537 tablet_copy_source_session.cc:205] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:04.299883 24230 catalog_manager.cc:4830] TS 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003): tablet 3071edf0c1004a6496d1cb46591d422e (table rebalance_test_table_3 [id=e6ccc802e346493aa6739ba0f901f14d]) successfully deleted
I20240503 08:12:04.303997 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c65da71faf34351885b5033d3d68835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:04.325695 26539 ts_tablet_manager.cc:1875] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:04.327343 24516 raft_consensus.cc:1261] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Refusing update from remote peer a0d221e07029452cae34b8920884507e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:12:04.329779 26552 consensus_queue.cc:1035] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:04.333145 25609 raft_consensus.cc:1050] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42: attempting to promote NON_VOTER 9d313001b0a647caba8f53540a2b67ac to VOTER
I20240503 08:12:04.335728 25609 consensus_queue.cc:237] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
I20240503 08:12:04.342576 24515 raft_consensus.cc:1261] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240503 08:12:04.343312 25960 raft_consensus.cc:1261] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240503 08:12:04.349810 26401 consensus_queue.cc:1035] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20240503 08:12:04.353245 26401 consensus_queue.cc:1035] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20240503 08:12:04.355152 26417 tablet_bootstrap.cc:492] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e: Bootstrap replayed 1/1 log segments. Stats: ops{read=67 overwritten=0 applied=67 ignored=0} inserts{seen=698 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:04.356021 26417 tablet_bootstrap.cc:492] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e: Bootstrap complete.
I20240503 08:12:04.356431 26417 ts_tablet_manager.cc:1367] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e: Time spent bootstrapping tablet: real 1.595s user 0.135s sys 0.014s
I20240503 08:12:04.365397 26400 raft_consensus.cc:2932] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index 70 to 88, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
W20240503 08:12:04.366598 26025 consensus_peers.cc:485] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e -> Peer 9d313001b0a647caba8f53540a2b67ac (127.20.107.134:46393): Couldn't send request to peer 9d313001b0a647caba8f53540a2b67ac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d422797de9843edbc473fca1d8e9e87. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:04.368249 26536 tablet_copy_client.cc:783] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:04.368592 26536 tablet_copy_client.cc:647] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:04.359004 26417 raft_consensus.cc:348] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } }
I20240503 08:12:04.372502 26536 tablet_copy_client.cc:523] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:04.374851 24516 raft_consensus.cc:2932] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } } }
I20240503 08:12:04.378046 24517 raft_consensus.cc:2932] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index 70 to 88, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:04.374125 26438 raft_consensus.cc:2932] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } } }
I20240503 08:12:04.388372 26417 raft_consensus.cc:729] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0d221e07029452cae34b8920884507e, State: Initialized, Role: LEARNER
I20240503 08:12:04.389914 25961 raft_consensus.cc:2932] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index 70 to 88, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:04.392980 24244 catalog_manager.cc:5484] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 reported cstate change: config changed from index -1 to 3, NON_VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) added. New cstate: current_term: 2 leader_uuid: "a0d221e07029452cae34b8920884507e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } } }
I20240503 08:12:04.388689 26417 consensus_queue.cc:260] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 67, Last appended: 1.67, Last appended by leader: 67, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } }
I20240503 08:12:04.396214 26417 ts_tablet_manager.cc:1398] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20240503 08:12:04.407541 26539 ts_tablet_manager.cc:1888] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.84
I20240503 08:12:04.414539 26539 log.cc:1196] T 83697ce9030f4981af717d18eb6757a4 P 56fb8e40a4da4b3794f07ad8137ecf42: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/83697ce9030f4981af717d18eb6757a4
I20240503 08:12:04.415787 24666 tablet_copy_service.cc:342] P 56fb8e40a4da4b3794f07ad8137ecf42: Request end of tablet copy session a0d221e07029452cae34b8920884507e-d73b0d5398cd484ba45515cf95cd5b14 received from {username='slave'} at 127.20.107.135:33925
I20240503 08:12:04.415947 24666 tablet_copy_service.cc:433] P 56fb8e40a4da4b3794f07ad8137ecf42: ending tablet copy session a0d221e07029452cae34b8920884507e-d73b0d5398cd484ba45515cf95cd5b14 on tablet d73b0d5398cd484ba45515cf95cd5b14 with peer a0d221e07029452cae34b8920884507e
I20240503 08:12:04.417091 24230 catalog_manager.cc:4830] TS 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003): tablet 83697ce9030f4981af717d18eb6757a4 (table rebalance_test_table_4 [id=72d93aae96774cf887b14b98c748dffc]) successfully deleted
I20240503 08:12:04.419404 26536 tablet_bootstrap.cc:492] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac: Bootstrap starting.
I20240503 08:12:04.403378 24243 catalog_manager.cc:5484] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: config changed from index 70 to 88, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:04.447829 26542 tablet_copy_client.cc:308] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:34003
I20240503 08:12:04.458205 24646 consensus_queue.cc:237] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 1.88, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
I20240503 08:12:04.461964 25960 raft_consensus.cc:1261] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch)
I20240503 08:12:04.462508 26329 consensus_queue.cc:1035] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20240503 08:12:04.473480 26401 raft_consensus.cc:2932] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.89: config changed from index 88 to 89, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:04.481127 24230 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7b858bcd6e1c469d9f19f142106908d3 with cas_config_opid_index 88: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:04.486280 24243 catalog_manager.cc:5484] T 7b858bcd6e1c469d9f19f142106908d3 P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: config changed from index 88 to 89, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:04.489531 25960 raft_consensus.cc:2932] T 7b858bcd6e1c469d9f19f142106908d3 P 9d313001b0a647caba8f53540a2b67ac [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index 88 to 89, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:04.507707 24666 tablet_copy_service.cc:140] P 56fb8e40a4da4b3794f07ad8137ecf42: Received BeginTabletCopySession request for tablet c85cf905ebfa4e7b91a0af9ed2fa77da from peer f9705fdde3854639ba85097a161f37e7 ({username='slave'} at 127.20.107.133:39409)
I20240503 08:12:04.507902 24666 tablet_copy_service.cc:161] P 56fb8e40a4da4b3794f07ad8137ecf42: Beginning new tablet copy session on tablet c85cf905ebfa4e7b91a0af9ed2fa77da from peer f9705fdde3854639ba85097a161f37e7 at {username='slave'} at 127.20.107.133:39409: session id = f9705fdde3854639ba85097a161f37e7-c85cf905ebfa4e7b91a0af9ed2fa77da
I20240503 08:12:04.510882 24666 tablet_copy_source_session.cc:205] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:04.511991 26412 ts_tablet_manager.cc:902] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7: Initiating tablet copy from peer 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:12:04.515645 26542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85cf905ebfa4e7b91a0af9ed2fa77da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:04.536434 25699 raft_consensus.cc:1203] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.61->[1.62-1.103] Dedup: 1.73->[1.74-1.103]
I20240503 08:12:04.547528 24496 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b858bcd6e1c469d9f19f142106908d3 with delete_type TABLET_DATA_TOMBSTONED (TS 6a640901590545eeb69e216d2399ea51 not found in new config with opid_index 89) from {username='slave'} at 127.0.0.1:42320
I20240503 08:12:04.549427 26542 tablet_copy_client.cc:783] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:04.551342 26542 tablet_copy_client.cc:647] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:04.579607 26412 tablet_copy_client.cc:308] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:34003
I20240503 08:12:04.583140 24667 tablet_copy_service.cc:140] P 56fb8e40a4da4b3794f07ad8137ecf42: Received BeginTabletCopySession request for tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0 from peer e7416e7c03a74a24b8fe2508111770d7 ({username='slave'} at 127.20.107.132:41209)
I20240503 08:12:04.583355 24667 tablet_copy_service.cc:161] P 56fb8e40a4da4b3794f07ad8137ecf42: Beginning new tablet copy session on tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0 from peer e7416e7c03a74a24b8fe2508111770d7 at {username='slave'} at 127.20.107.132:41209: session id = e7416e7c03a74a24b8fe2508111770d7-5dd5613d173f4b1eb0eacf2fec3a3ff0
I20240503 08:12:04.586198 24667 tablet_copy_source_session.cc:205] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:04.587878 26542 tablet_copy_client.cc:523] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:04.594231 26412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:04.599207 26542 tablet_bootstrap.cc:492] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7: Bootstrap starting.
I20240503 08:12:04.602934 26569 tablet_replica.cc:330] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51: stopping tablet replica
I20240503 08:12:04.616848 26569 raft_consensus.cc:2229] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:04.618499 26569 raft_consensus.cc:2258] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:04.630838 26569 ts_tablet_manager.cc:1875] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:04.675274 26412 tablet_copy_client.cc:783] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:04.686780 26412 tablet_copy_client.cc:647] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:04.694334 26412 tablet_copy_client.cc:523] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:04.704082 26569 ts_tablet_manager.cc:1888] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.88
I20240503 08:12:04.708122 26412 tablet_bootstrap.cc:492] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7: Bootstrap starting.
I20240503 08:12:04.715734 26569 log.cc:1196] T 7b858bcd6e1c469d9f19f142106908d3 P 6a640901590545eeb69e216d2399ea51: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/7b858bcd6e1c469d9f19f142106908d3
I20240503 08:12:04.749507 24230 catalog_manager.cc:4830] TS 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793): tablet 7b858bcd6e1c469d9f19f142106908d3 (table rebalance_test_table_1 [id=bb0497226b9c4da186bef24f42f0762a]) successfully deleted
I20240503 08:12:04.828362 25609 raft_consensus.cc:1050] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42: attempting to promote NON_VOTER e7416e7c03a74a24b8fe2508111770d7 to VOTER
I20240503 08:12:04.830359 25609 consensus_queue.cc:237] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 1.111, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } }
I20240503 08:12:04.831938 24384 raft_consensus.cc:1261] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 112. (index mismatch)
I20240503 08:12:04.832693 25609 consensus_queue.cc:1035] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20240503 08:12:04.835101 26401 raft_consensus.cc:2932] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.112: config changed from index 62 to 112, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:04.841341 24244 catalog_manager.cc:5484] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: config changed from index 62 to 112, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:04.844267 26487 tablet_bootstrap.cc:492] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e: Bootstrap replayed 1/1 log segments. Stats: ops{read=99 overwritten=0 applied=98 ignored=0} inserts{seen=783 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:12:04.846534 24384 raft_consensus.cc:2932] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.112: config changed from index 62 to 112, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:04.850605 26487 tablet_bootstrap.cc:492] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e: Bootstrap complete.
I20240503 08:12:04.850992 26487 ts_tablet_manager.cc:1367] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e: Time spent bootstrapping tablet: real 1.175s user 0.167s sys 0.008s
I20240503 08:12:04.854022 26487 raft_consensus.cc:348] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } }
I20240503 08:12:04.869333 25697 raft_consensus.cc:1261] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 112. (index mismatch)
I20240503 08:12:04.885064 26487 raft_consensus.cc:729] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0d221e07029452cae34b8920884507e, State: Initialized, Role: LEARNER
I20240503 08:12:04.885493 26487 consensus_queue.cc:260] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 98, Last appended: 1.99, Last appended by leader: 99, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } }
I20240503 08:12:04.886194 26487 ts_tablet_manager.cc:1398] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20240503 08:12:04.887063 24537 tablet_copy_service.cc:342] P 6a640901590545eeb69e216d2399ea51: Request end of tablet copy session a0d221e07029452cae34b8920884507e-61bf5a9f26034821b821a45ac37ba133 received from {username='slave'} at 127.20.107.135:39019
I20240503 08:12:04.887209 24537 tablet_copy_service.cc:433] P 6a640901590545eeb69e216d2399ea51: ending tablet copy session a0d221e07029452cae34b8920884507e-61bf5a9f26034821b821a45ac37ba133 on tablet 61bf5a9f26034821b821a45ac37ba133 with peer a0d221e07029452cae34b8920884507e
I20240503 08:12:04.897852 26093 raft_consensus.cc:1203] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.55->[1.56-1.98] Dedup: 1.67->[1.68-1.98]
I20240503 08:12:04.900357 26329 consensus_queue.cc:1035] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20240503 08:12:04.902482 25697 raft_consensus.cc:2932] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Committing config change with OpId 1.112: config changed from index 62 to 112, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:04.907330 26583 ts_tablet_manager.cc:902] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac: Initiating tablet copy from peer a0d221e07029452cae34b8920884507e (127.20.107.135:45329)
I20240503 08:12:04.936206 24646 consensus_queue.cc:237] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 1.115, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } }
I20240503 08:12:04.938400 25697 raft_consensus.cc:1261] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 114. Preceding OpId from leader: term: 1 index: 116. (index mismatch)
I20240503 08:12:04.939538 25609 consensus_queue.cc:1035] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 114, Time since last communication: 0.000s
I20240503 08:12:04.947002 26252 raft_consensus.cc:1050] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42: attempting to promote NON_VOTER a0d221e07029452cae34b8920884507e to VOTER
I20240503 08:12:04.947916 26252 consensus_queue.cc:237] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 1.99, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } }
I20240503 08:12:04.949826 24383 raft_consensus.cc:1261] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 99. Preceding OpId from leader: term: 1 index: 100. (index mismatch)
I20240503 08:12:04.949880 26092 raft_consensus.cc:1261] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 1 LEARNER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 99. Preceding OpId from leader: term: 1 index: 100. (index mismatch)
I20240503 08:12:04.951320 25609 consensus_queue.cc:1035] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20240503 08:12:04.952860 26400 consensus_queue.cc:1035] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.001s
I20240503 08:12:04.955857 24386 raft_consensus.cc:2932] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.100: config changed from index 56 to 100, a0d221e07029452cae34b8920884507e (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:04.962563 24244 catalog_manager.cc:5484] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: config changed from index 56 to 100, a0d221e07029452cae34b8920884507e (127.20.107.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:04.966619 26329 raft_consensus.cc:2932] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.100: config changed from index 56 to 100, a0d221e07029452cae34b8920884507e (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:04.967319 26092 raft_consensus.cc:2932] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Committing config change with OpId 1.100: config changed from index 56 to 100, a0d221e07029452cae34b8920884507e (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:04.996737 26420 tablet_bootstrap.cc:492] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=78 overwritten=0 applied=78 ignored=0} inserts{seen=750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:05.015534 26420 tablet_bootstrap.cc:492] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac: Bootstrap complete.
I20240503 08:12:05.015901 26420 ts_tablet_manager.cc:1367] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac: Time spent bootstrapping tablet: real 1.132s user 0.139s sys 0.010s
I20240503 08:12:05.016484 24646 consensus_queue.cc:237] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 1.101, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } }
I20240503 08:12:05.019765 26092 raft_consensus.cc:1261] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240503 08:12:05.020584 26401 consensus_queue.cc:1035] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20240503 08:12:05.023773 26401 raft_consensus.cc:2932] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.102: config changed from index 100 to 102, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:05.024535 26092 raft_consensus.cc:2932] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 100 to 102, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:05.030884 26252 raft_consensus.cc:2932] T 7d2772d08c06498db2cb0b02497ea042 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.116: config changed from index 112 to 116, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:05.032728 24244 catalog_manager.cc:5484] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e reported cstate change: config changed from index 100 to 102, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:05.034860 24230 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d73b0d5398cd484ba45515cf95cd5b14 with cas_config_opid_index 100: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:05.036710 24230 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d2772d08c06498db2cb0b02497ea042 with cas_config_opid_index 112: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:05.037586 25697 raft_consensus.cc:2932] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Committing config change with OpId 1.116: config changed from index 112 to 116, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:05.043375 24240 catalog_manager.cc:5484] T 7d2772d08c06498db2cb0b02497ea042 P e7416e7c03a74a24b8fe2508111770d7 reported cstate change: config changed from index 112 to 116, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:05.044819 26420 raft_consensus.cc:348] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } }
I20240503 08:12:05.045214 26420 raft_consensus.cc:729] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d313001b0a647caba8f53540a2b67ac, State: Initialized, Role: LEARNER
I20240503 08:12:05.045430 26420 consensus_queue.cc:260] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 78, Last appended: 1.78, Last appended by leader: 78, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } }
I20240503 08:12:05.049356 24364 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d2772d08c06498db2cb0b02497ea042 with delete_type TABLET_DATA_TOMBSTONED (TS b53502e2a1ae4c86a054d38048dd0df6 not found in new config with opid_index 116) from {username='slave'} at 127.0.0.1:38614
I20240503 08:12:05.051877 24366 tablet_service.cc:1514] Processing DeleteTablet for tablet d73b0d5398cd484ba45515cf95cd5b14 with delete_type TABLET_DATA_TOMBSTONED (TS b53502e2a1ae4c86a054d38048dd0df6 not found in new config with opid_index 102) from {username='slave'} at 127.0.0.1:38614
I20240503 08:12:05.056654 26420 ts_tablet_manager.cc:1398] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac: Time spent starting tablet: real 0.041s user 0.003s sys 0.000s
I20240503 08:12:05.057516 24537 tablet_copy_service.cc:342] P 6a640901590545eeb69e216d2399ea51: Request end of tablet copy session 9d313001b0a647caba8f53540a2b67ac-d8a32af7adaf410582e1b898257db62d received from {username='slave'} at 127.20.107.134:56731
I20240503 08:12:05.057684 24537 tablet_copy_service.cc:433] P 6a640901590545eeb69e216d2399ea51: ending tablet copy session 9d313001b0a647caba8f53540a2b67ac-d8a32af7adaf410582e1b898257db62d on tablet d8a32af7adaf410582e1b898257db62d with peer 9d313001b0a647caba8f53540a2b67ac
I20240503 08:12:05.075350 26597 tablet_replica.cc:330] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6: stopping tablet replica
I20240503 08:12:05.077769 26597 raft_consensus.cc:2229] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:05.078841 26597 raft_consensus.cc:2258] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:05.087810 26597 ts_tablet_manager.cc:1875] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:05.099922 26092 raft_consensus.cc:1203] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.88->[1.89-1.109] Dedup: 1.99->[1.100-1.109]
I20240503 08:12:05.121428 26583 tablet_copy_client.cc:308] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.135:45329
I20240503 08:12:05.120848 26597 ts_tablet_manager.cc:1888] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.115
I20240503 08:12:05.132647 26597 log.cc:1196] T 7d2772d08c06498db2cb0b02497ea042 P b53502e2a1ae4c86a054d38048dd0df6: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/7d2772d08c06498db2cb0b02497ea042
I20240503 08:12:05.134469 26597 tablet_replica.cc:330] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6: stopping tablet replica
I20240503 08:12:05.134698 24229 catalog_manager.cc:4830] TS b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775): tablet 7d2772d08c06498db2cb0b02497ea042 (table rebalance_test_table_2 [id=4f8b8cf1869a436f9c012044eb10dc54]) successfully deleted
I20240503 08:12:05.140412 26597 raft_consensus.cc:2229] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:05.141858 26597 raft_consensus.cc:2258] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:05.175099 26113 tablet_copy_service.cc:140] P a0d221e07029452cae34b8920884507e: Received BeginTabletCopySession request for tablet 6d422797de9843edbc473fca1d8e9e87 from peer 9d313001b0a647caba8f53540a2b67ac ({username='slave'} at 127.20.107.134:36721)
I20240503 08:12:05.175419 26113 tablet_copy_service.cc:161] P a0d221e07029452cae34b8920884507e: Beginning new tablet copy session on tablet 6d422797de9843edbc473fca1d8e9e87 from peer 9d313001b0a647caba8f53540a2b67ac at {username='slave'} at 127.20.107.134:36721: session id = 9d313001b0a647caba8f53540a2b67ac-6d422797de9843edbc473fca1d8e9e87
I20240503 08:12:05.178834 26113 tablet_copy_source_session.cc:205] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:05.180478 26583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d422797de9843edbc473fca1d8e9e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:05.191325 26583 tablet_copy_client.cc:783] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:05.191723 26583 tablet_copy_client.cc:647] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:05.191910 26597 ts_tablet_manager.cc:1875] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:05.193814 26583 tablet_copy_client.cc:523] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:05.202275 26583 tablet_bootstrap.cc:492] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac: Bootstrap starting.
I20240503 08:12:05.207864 26345 raft_consensus.cc:1050] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51: attempting to promote NON_VOTER a0d221e07029452cae34b8920884507e to VOTER
I20240503 08:12:05.208564 26345 consensus_queue.cc:237] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 1.110, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } }
I20240503 08:12:05.210206 26092 raft_consensus.cc:1261] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e [term 1 LEARNER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240503 08:12:05.211494 26558 consensus_queue.cc:1035] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 109, Time since last communication: 0.000s
I20240503 08:12:05.213424 24646 raft_consensus.cc:1261] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240503 08:12:05.214108 26558 raft_consensus.cc:2932] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.111: config changed from index 89 to 111, a0d221e07029452cae34b8920884507e (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:05.214696 26092 raft_consensus.cc:2932] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index 89 to 111, a0d221e07029452cae34b8920884507e (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:05.219134 26558 consensus_queue.cc:1035] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 109, Time since last communication: 0.000s
I20240503 08:12:05.220613 24244 catalog_manager.cc:5484] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 reported cstate change: config changed from index 89 to 111, a0d221e07029452cae34b8920884507e (127.20.107.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:05.226444 24517 consensus_queue.cc:237] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 1.111, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } }
I20240503 08:12:05.228732 26092 raft_consensus.cc:1261] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 112. (index mismatch)
I20240503 08:12:05.229452 26558 consensus_queue.cc:1035] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20240503 08:12:05.232486 25529 raft_consensus.cc:2932] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.112: config changed from index 111 to 112, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:05.237521 24230 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 61bf5a9f26034821b821a45ac37ba133 with cas_config_opid_index 111: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:05.239542 24243 catalog_manager.cc:5484] T 61bf5a9f26034821b821a45ac37ba133 P 6a640901590545eeb69e216d2399ea51 reported cstate change: config changed from index 111 to 112, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:05.243595 24627 tablet_service.cc:1514] Processing DeleteTablet for tablet 61bf5a9f26034821b821a45ac37ba133 with delete_type TABLET_DATA_TOMBSTONED (TS 56fb8e40a4da4b3794f07ad8137ecf42 not found in new config with opid_index 112) from {username='slave'} at 127.0.0.1:46200
I20240503 08:12:05.245426 26092 raft_consensus.cc:2932] T 61bf5a9f26034821b821a45ac37ba133 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Committing config change with OpId 1.112: config changed from index 111 to 112, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:05.255784 24646 raft_consensus.cc:2932] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index 89 to 111, a0d221e07029452cae34b8920884507e (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:05.261960 26605 tablet_replica.cc:330] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42: stopping tablet replica
I20240503 08:12:05.262437 26605 raft_consensus.cc:2229] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:05.262853 26605 raft_consensus.cc:2258] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:05.264032 26605 ts_tablet_manager.cc:1875] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:05.273644 26412 tablet_bootstrap.cc:492] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=94 overwritten=0 applied=94 ignored=0} inserts{seen=725 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:05.274237 26412 tablet_bootstrap.cc:492] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7: Bootstrap complete.
I20240503 08:12:05.274631 26412 ts_tablet_manager.cc:1367] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7: Time spent bootstrapping tablet: real 0.567s user 0.158s sys 0.004s
I20240503 08:12:05.275475 26412 raft_consensus.cc:348] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } }
I20240503 08:12:05.275789 26412 raft_consensus.cc:729] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e7416e7c03a74a24b8fe2508111770d7, State: Initialized, Role: LEARNER
I20240503 08:12:05.276036 26412 consensus_queue.cc:260] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 94, Last appended: 1.94, Last appended by leader: 94, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } }
I20240503 08:12:05.276700 26412 ts_tablet_manager.cc:1398] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:05.277611 24667 tablet_copy_service.cc:342] P 56fb8e40a4da4b3794f07ad8137ecf42: Request end of tablet copy session e7416e7c03a74a24b8fe2508111770d7-5dd5613d173f4b1eb0eacf2fec3a3ff0 received from {username='slave'} at 127.20.107.132:41209
I20240503 08:12:05.277750 24667 tablet_copy_service.cc:433] P 56fb8e40a4da4b3794f07ad8137ecf42: ending tablet copy session e7416e7c03a74a24b8fe2508111770d7-5dd5613d173f4b1eb0eacf2fec3a3ff0 on tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0 with peer e7416e7c03a74a24b8fe2508111770d7
I20240503 08:12:05.287941 25960 raft_consensus.cc:1203] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Deduplicated request from leader. Original: 1.70->[1.71-1.100] Dedup: 1.78->[1.79-1.100]
I20240503 08:12:05.319765 26536 tablet_bootstrap.cc:492] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=91 overwritten=0 applied=91 ignored=0} inserts{seen=763 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:05.320377 26536 tablet_bootstrap.cc:492] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac: Bootstrap complete.
I20240503 08:12:05.324347 26536 ts_tablet_manager.cc:1367] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac: Time spent bootstrapping tablet: real 0.905s user 0.164s sys 0.006s
I20240503 08:12:05.327433 26605 ts_tablet_manager.cc:1888] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.111
I20240503 08:12:05.332134 26536 raft_consensus.cc:348] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } }
I20240503 08:12:05.332314 26605 log.cc:1196] T 61bf5a9f26034821b821a45ac37ba133 P 56fb8e40a4da4b3794f07ad8137ecf42: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/61bf5a9f26034821b821a45ac37ba133
I20240503 08:12:05.332557 26536 raft_consensus.cc:729] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d313001b0a647caba8f53540a2b67ac, State: Initialized, Role: LEARNER
I20240503 08:12:05.332834 26536 consensus_queue.cc:260] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 91, Last appended: 1.91, Last appended by leader: 91, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } }
I20240503 08:12:05.333676 26536 ts_tablet_manager.cc:1398] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:05.333804 24230 catalog_manager.cc:4830] TS 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003): tablet 61bf5a9f26034821b821a45ac37ba133 (table rebalance_test_table_1 [id=bb0497226b9c4da186bef24f42f0762a]) successfully deleted
I20240503 08:12:05.334451 24537 tablet_copy_service.cc:342] P 6a640901590545eeb69e216d2399ea51: Request end of tablet copy session 9d313001b0a647caba8f53540a2b67ac-3c65da71faf34351885b5033d3d68835 received from {username='slave'} at 127.20.107.134:56731
I20240503 08:12:05.334625 24537 tablet_copy_service.cc:433] P 6a640901590545eeb69e216d2399ea51: ending tablet copy session 9d313001b0a647caba8f53540a2b67ac-3c65da71faf34351885b5033d3d68835 on tablet 3c65da71faf34351885b5033d3d68835 with peer 9d313001b0a647caba8f53540a2b67ac
I20240503 08:12:05.381901 26542 tablet_bootstrap.cc:492] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=110 overwritten=0 applied=110 ignored=0} inserts{seen=795 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:05.382581 26542 tablet_bootstrap.cc:492] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7: Bootstrap complete.
I20240503 08:12:05.383088 26542 ts_tablet_manager.cc:1367] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7: Time spent bootstrapping tablet: real 0.784s user 0.186s sys 0.012s
I20240503 08:12:05.384586 26542 raft_consensus.cc:348] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } }
I20240503 08:12:05.385346 26542 raft_consensus.cc:729] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f9705fdde3854639ba85097a161f37e7, State: Initialized, Role: LEARNER
I20240503 08:12:05.385658 26542 consensus_queue.cc:260] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 110, Last appended: 1.110, Last appended by leader: 110, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } }
I20240503 08:12:05.397511 26542 ts_tablet_manager.cc:1398] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7: Time spent starting tablet: real 0.014s user 0.007s sys 0.004s
I20240503 08:12:05.398352 24667 tablet_copy_service.cc:342] P 56fb8e40a4da4b3794f07ad8137ecf42: Request end of tablet copy session f9705fdde3854639ba85097a161f37e7-c85cf905ebfa4e7b91a0af9ed2fa77da received from {username='slave'} at 127.20.107.133:39409
I20240503 08:12:05.398485 24667 tablet_copy_service.cc:433] P 56fb8e40a4da4b3794f07ad8137ecf42: ending tablet copy session f9705fdde3854639ba85097a161f37e7-c85cf905ebfa4e7b91a0af9ed2fa77da on tablet c85cf905ebfa4e7b91a0af9ed2fa77da with peer f9705fdde3854639ba85097a161f37e7
I20240503 08:12:05.416956 25529 raft_consensus.cc:1050] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51: attempting to promote NON_VOTER 9d313001b0a647caba8f53540a2b67ac to VOTER
I20240503 08:12:05.417966 25529 consensus_queue.cc:237] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 1.101, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
I20240503 08:12:05.420964 25961 raft_consensus.cc:1261] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240503 08:12:05.421696 26345 consensus_queue.cc:1035] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20240503 08:12:05.422726 24383 raft_consensus.cc:1261] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240503 08:12:05.425132 26345 raft_consensus.cc:2932] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.102: config changed from index 71 to 102, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:05.433462 25961 raft_consensus.cc:2932] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 71 to 102, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:05.435425 26345 consensus_queue.cc:1035] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20240503 08:12:05.432569 24243 catalog_manager.cc:5484] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 reported cstate change: config changed from index 71 to 102, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:05.438855 26583 tablet_bootstrap.cc:492] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:05.445994 26597 ts_tablet_manager.cc:1888] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.101
I20240503 08:12:05.446225 26597 log.cc:1196] T d73b0d5398cd484ba45515cf95cd5b14 P b53502e2a1ae4c86a054d38048dd0df6: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/d73b0d5398cd484ba45515cf95cd5b14
I20240503 08:12:05.447212 24229 catalog_manager.cc:4830] TS b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775): tablet d73b0d5398cd484ba45515cf95cd5b14 (table rebalance_test_table_4 [id=72d93aae96774cf887b14b98c748dffc]) successfully deleted
I20240503 08:12:05.454141 24517 consensus_queue.cc:237] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
I20240503 08:12:05.457137 25960 raft_consensus.cc:1261] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240503 08:12:05.457253 26583 tablet_bootstrap.cc:492] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac: Bootstrap complete.
I20240503 08:12:05.457686 26583 ts_tablet_manager.cc:1367] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac: Time spent bootstrapping tablet: real 0.256s user 0.041s sys 0.014s
I20240503 08:12:05.457640 26345 consensus_queue.cc:1035] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 102, Time since last communication: 0.000s
I20240503 08:12:05.459589 26583 raft_consensus.cc:348] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } }
I20240503 08:12:05.461249 26583 raft_consensus.cc:729] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9d313001b0a647caba8f53540a2b67ac, State: Initialized, Role: LEARNER
I20240503 08:12:05.462199 26345 raft_consensus.cc:2932] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index 102 to 104, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:05.461587 26583 consensus_queue.cc:260] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: true } }
I20240503 08:12:05.467537 24230 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d8a32af7adaf410582e1b898257db62d with cas_config_opid_index 102: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:05.469508 24244 catalog_manager.cc:5484] T d8a32af7adaf410582e1b898257db62d P 6a640901590545eeb69e216d2399ea51 reported cstate change: config changed from index 102 to 104, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:05.471002 24384 raft_consensus.cc:2932] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 71 to 102, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:05.475631 25960 raft_consensus.cc:1203] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Deduplicated request from leader. Original: 1.85->[1.86-1.102] Dedup: 1.91->[1.92-1.102]
I20240503 08:12:05.478425 24364 tablet_service.cc:1514] Processing DeleteTablet for tablet d8a32af7adaf410582e1b898257db62d with delete_type TABLET_DATA_TOMBSTONED (TS b53502e2a1ae4c86a054d38048dd0df6 not found in new config with opid_index 104) from {username='slave'} at 127.0.0.1:38614
I20240503 08:12:05.481181 26597 tablet_replica.cc:330] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6: stopping tablet replica
I20240503 08:12:05.481410 26597 raft_consensus.cc:2229] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:05.482213 26597 pending_rounds.cc:62] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6: Trying to abort 1 pending ops.
I20240503 08:12:05.485440 25961 raft_consensus.cc:2932] T d8a32af7adaf410582e1b898257db62d P 9d313001b0a647caba8f53540a2b67ac [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 102 to 104, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:05.494941 26583 ts_tablet_manager.cc:1398] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac: Time spent starting tablet: real 0.036s user 0.000s sys 0.002s
I20240503 08:12:05.482342 26597 pending_rounds.cc:68] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6: Aborting op as it isn't in flight: id { term: 1 index: 103 } timestamp: 7023509198551318528 op_type: WRITE_OP write_request { tablet_id: "d8a32af7adaf410582e1b898257db62d" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\211\341\315J\321JG\263\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""0742c274dfd8e471" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023509198468976640 authz_token { token_data: "\010\372\272\322\261\006\"3\n\005slave\022*\n 72d93aae96774cf887b14b98c748dffc\020\001\030\001 \001(\001" signature: "<redacted>""s]\267\004d\245\2703\2440\346\023\263\267\016xY\327\270\304\037\'\311D\271u\'\372\306\336\367b\001R_r\300Wco\341\257\227\003\356\251\364<{sv\363\373\227\305\353/\375\232\014k\235?\314" signing_key_seq_num: 0 } } request_id { client_id: "9a9c7e884a454d8ab3339ea71aa1d244" seq_no: 293 first_incomplete_seq_no: 293 attempt_no: 0 }
I20240503 08:12:05.498731 26597 raft_consensus.cc:2258] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:05.502143 26113 tablet_copy_service.cc:342] P a0d221e07029452cae34b8920884507e: Request end of tablet copy session 9d313001b0a647caba8f53540a2b67ac-6d422797de9843edbc473fca1d8e9e87 received from {username='slave'} at 127.20.107.134:36721
I20240503 08:12:05.502306 26113 tablet_copy_service.cc:433] P a0d221e07029452cae34b8920884507e: ending tablet copy session 9d313001b0a647caba8f53540a2b67ac-6d422797de9843edbc473fca1d8e9e87 on tablet 6d422797de9843edbc473fca1d8e9e87 with peer 9d313001b0a647caba8f53540a2b67ac
I20240503 08:12:05.559250 26597 ts_tablet_manager.cc:1875] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:05.588790 26597 ts_tablet_manager.cc:1888] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.103
I20240503 08:12:05.598683 25697 raft_consensus.cc:1203] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.87->[1.88-1.101] Dedup: 1.94->[1.95-1.101]
I20240503 08:12:05.599021 26597 log.cc:1196] T d8a32af7adaf410582e1b898257db62d P b53502e2a1ae4c86a054d38048dd0df6: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/d8a32af7adaf410582e1b898257db62d
I20240503 08:12:05.599988 24229 catalog_manager.cc:4830] TS b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775): tablet d8a32af7adaf410582e1b898257db62d (table rebalance_test_table_4 [id=72d93aae96774cf887b14b98c748dffc]) successfully deleted
I20240503 08:12:05.648092 26400 raft_consensus.cc:1050] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42: attempting to promote NON_VOTER e7416e7c03a74a24b8fe2508111770d7 to VOTER
I20240503 08:12:05.648886 26400 consensus_queue.cc:237] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.102, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } }
I20240503 08:12:05.650965 24386 raft_consensus.cc:1261] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240503 08:12:05.651578 26401 consensus_queue.cc:1035] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240503 08:12:05.653128 25699 raft_consensus.cc:1261] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240503 08:12:05.654942 26329 consensus_queue.cc:1035] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 101, Time since last communication: 0.000s
I20240503 08:12:05.660032 26329 raft_consensus.cc:2932] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.103: config changed from index 88 to 103, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:05.667980 24384 raft_consensus.cc:2932] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 88 to 103, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:05.675922 24243 catalog_manager.cc:5484] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: config changed from index 88 to 103, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:05.698590 25697 raft_consensus.cc:2932] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 88 to 103, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:05.699407 25529 raft_consensus.cc:1050] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51: attempting to promote NON_VOTER 9d313001b0a647caba8f53540a2b67ac to VOTER
I20240503 08:12:05.705089 25529 consensus_queue.cc:237] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
I20240503 08:12:05.707887 24384 raft_consensus.cc:1261] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240503 08:12:05.708815 25960 raft_consensus.cc:1261] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac [term 1 LEARNER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240503 08:12:05.710963 25529 consensus_queue.cc:1035] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240503 08:12:05.711921 26558 consensus_queue.cc:1035] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240503 08:12:05.714290 26237 raft_consensus.cc:2932] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index 86 to 104, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:05.714859 24386 raft_consensus.cc:2932] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 86 to 104, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:05.720607 24244 catalog_manager.cc:5484] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: config changed from index 86 to 104, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:05.725951 25830 raft_consensus.cc:1203] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.98->[1.99-1.118] Dedup: 1.110->[1.111-1.118]
I20240503 08:12:05.738967 25961 raft_consensus.cc:2932] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 86 to 104, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:05.829633 26628 raft_consensus.cc:1050] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42: attempting to promote NON_VOTER f9705fdde3854639ba85097a161f37e7 to VOTER
I20240503 08:12:05.833038 26628 consensus_queue.cc:237] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 1.119, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } }
I20240503 08:12:05.845142 24384 raft_consensus.cc:1261] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 119. Preceding OpId from leader: term: 1 index: 120. (index mismatch)
I20240503 08:12:05.846311 25830 raft_consensus.cc:1261] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7 [term 1 LEARNER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 119. Preceding OpId from leader: term: 1 index: 120. (index mismatch)
I20240503 08:12:05.848111 26329 consensus_queue.cc:1035] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20240503 08:12:05.850677 26628 consensus_queue.cc:1035] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20240503 08:12:05.861850 25831 raft_consensus.cc:2932] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7 [term 1 FOLLOWER]: Committing config change with OpId 1.120: config changed from index 99 to 120, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:05.863224 24383 raft_consensus.cc:2932] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.120: config changed from index 99 to 120, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:05.878494 24240 catalog_manager.cc:5484] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: config changed from index 99 to 120, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:05.912499 26401 raft_consensus.cc:2932] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.120: config changed from index 99 to 120, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:05.923916 24645 consensus_queue.cc:237] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 1.120, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } }
I20240503 08:12:05.929620 25830 raft_consensus.cc:1261] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7 [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20240503 08:12:05.933612 26634 consensus_queue.cc:1035] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20240503 08:12:05.937906 26400 raft_consensus.cc:2932] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.121: config changed from index 120 to 121, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:05.945184 25830 raft_consensus.cc:2932] T c85cf905ebfa4e7b91a0af9ed2fa77da P f9705fdde3854639ba85097a161f37e7 [term 1 FOLLOWER]: Committing config change with OpId 1.121: config changed from index 120 to 121, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:05.945838 24230 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c85cf905ebfa4e7b91a0af9ed2fa77da with cas_config_opid_index 120: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:05.949575 24244 catalog_manager.cc:5484] T c85cf905ebfa4e7b91a0af9ed2fa77da P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: config changed from index 120 to 121, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:05.953912 24364 tablet_service.cc:1514] Processing DeleteTablet for tablet c85cf905ebfa4e7b91a0af9ed2fa77da with delete_type TABLET_DATA_TOMBSTONED (TS b53502e2a1ae4c86a054d38048dd0df6 not found in new config with opid_index 121) from {username='slave'} at 127.0.0.1:38614
I20240503 08:12:05.954317 26597 tablet_replica.cc:330] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6: stopping tablet replica
I20240503 08:12:05.954517 26597 raft_consensus.cc:2229] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:05.955181 26597 raft_consensus.cc:2258] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:05.956249 26597 ts_tablet_manager.cc:1875] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:05.981804 25961 raft_consensus.cc:1203] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240503 08:12:05.998453 26597 ts_tablet_manager.cc:1888] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.120
I20240503 08:12:05.998664 26597 log.cc:1196] T c85cf905ebfa4e7b91a0af9ed2fa77da P b53502e2a1ae4c86a054d38048dd0df6: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/c85cf905ebfa4e7b91a0af9ed2fa77da
I20240503 08:12:06.001331 24229 catalog_manager.cc:4830] TS b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775): tablet c85cf905ebfa4e7b91a0af9ed2fa77da (table rebalance_test_table_0 [id=492713985fb14342af075b33cdc4b9af]) successfully deleted
I20240503 08:12:06.171345 24516 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e0756a24eb5d4e26b2d44e46638a74be"
dest_uuid: "6a640901590545eeb69e216d2399ea51"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44650
I20240503 08:12:06.175359 24516 raft_consensus.cc:595] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Received request to transfer leadership
W20240503 08:12:06.213585 24474 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Service unavailable: leader transfer in progress
W20240503 08:12:06.224449 24474 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Service unavailable: leader transfer in progress
W20240503 08:12:06.244072 24474 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Service unavailable: leader transfer in progress
W20240503 08:12:06.252918 24474 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Service unavailable: leader transfer in progress
W20240503 08:12:06.261255 24474 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Service unavailable: leader transfer in progress
W20240503 08:12:06.272239 24474 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Service unavailable: leader transfer in progress
W20240503 08:12:06.281172 24474 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Service unavailable: leader transfer in progress
W20240503 08:12:06.294090 24474 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Service unavailable: leader transfer in progress
W20240503 08:12:06.306998 24474 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Service unavailable: leader transfer in progress
W20240503 08:12:06.319047 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Service unavailable: leader transfer in progress
W20240503 08:12:06.333707 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Service unavailable: leader transfer in progress
W20240503 08:12:06.349081 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Service unavailable: leader transfer in progress
W20240503 08:12:06.369109 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Service unavailable: leader transfer in progress
W20240503 08:12:06.386062 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Service unavailable: leader transfer in progress
I20240503 08:12:06.406944 26182 raft_consensus.cc:979] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51: : Instructing follower f9705fdde3854639ba85097a161f37e7 to start an election
I20240503 08:12:06.407160 26182 raft_consensus.cc:1067] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Signalling peer f9705fdde3854639ba85097a161f37e7 to start an election
I20240503 08:12:06.409372 25830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e0756a24eb5d4e26b2d44e46638a74be"
dest_uuid: "f9705fdde3854639ba85097a161f37e7"
from {username='slave'} at 127.20.107.130:43827
I20240503 08:12:06.409710 25830 raft_consensus.cc:482] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:06.409888 25830 raft_consensus.cc:3037] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:12:06.415221 24474 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Service unavailable: leader transfer in progress
I20240503 08:12:06.415673 25830 raft_consensus.cc:504] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } }
I20240503 08:12:06.419759 24516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0756a24eb5d4e26b2d44e46638a74be" candidate_uuid: "f9705fdde3854639ba85097a161f37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 110 } } ignore_live_leader: true dest_uuid: "6a640901590545eeb69e216d2399ea51"
I20240503 08:12:06.419986 24516 raft_consensus.cc:3032] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:06.420105 24516 raft_consensus.cc:729] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Running, Role: LEADER
I20240503 08:12:06.420433 24516 consensus_queue.cc:260] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.110, Last appended by leader: 110, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } }
I20240503 08:12:06.421015 24516 raft_consensus.cc:3037] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:06.421779 24646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0756a24eb5d4e26b2d44e46638a74be" candidate_uuid: "f9705fdde3854639ba85097a161f37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 110 } } ignore_live_leader: true dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42"
I20240503 08:12:06.422013 24646 raft_consensus.cc:3037] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:06.424736 24516 raft_consensus.cc:2454] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9705fdde3854639ba85097a161f37e7 in term 2.
I20240503 08:12:06.424742 24646 raft_consensus.cc:2454] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9705fdde3854639ba85097a161f37e7 in term 2.
I20240503 08:12:06.425987 25765 leader_election.cc:302] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, f9705fdde3854639ba85097a161f37e7; no voters:
I20240503 08:12:06.426949 26609 raft_consensus.cc:2781] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:06.427878 26609 raft_consensus.cc:686] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [term 2 LEADER]: Becoming Leader. State: Replica: f9705fdde3854639ba85097a161f37e7, State: Running, Role: LEADER
I20240503 08:12:06.428400 26609 consensus_queue.cc:237] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.110, Last appended by leader: 110, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } }
I20240503 08:12:06.425726 25830 leader_election.cc:288] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [CANDIDATE]: Term 2 election: Requested vote from peers 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003), 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:12:06.432735 24240 catalog_manager.cc:5484] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 reported cstate change: term changed from 1 to 2, leader changed from 6a640901590545eeb69e216d2399ea51 (127.20.107.130) to f9705fdde3854639ba85097a161f37e7 (127.20.107.133). New cstate: current_term: 2 leader_uuid: "f9705fdde3854639ba85097a161f37e7" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:12:06.436336 24474 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Illegal state: replica 6a640901590545eeb69e216d2399ea51 is not leader of this config: current role FOLLOWER
I20240503 08:12:06.540499 24646 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08df2ee2ca344cd2abd5ecb67ba9ffc1"
dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39514
I20240503 08:12:06.540799 24646 raft_consensus.cc:595] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:06.591573 26438 raft_consensus.cc:1050] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: attempting to promote NON_VOTER 9d313001b0a647caba8f53540a2b67ac to VOTER
I20240503 08:12:06.595583 26438 consensus_queue.cc:237] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
W20240503 08:12:06.603677 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
I20240503 08:12:06.609612 24516 raft_consensus.cc:1261] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Refusing update from remote peer a0d221e07029452cae34b8920884507e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:12:06.609750 25959 raft_consensus.cc:1261] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac [term 2 LEARNER]: Refusing update from remote peer a0d221e07029452cae34b8920884507e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:12:06.610530 26685 consensus_queue.cc:1035] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:12:06.611064 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
I20240503 08:12:06.611140 26685 consensus_queue.cc:1035] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:12:06.618352 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
W20240503 08:12:06.625312 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
W20240503 08:12:06.633270 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
W20240503 08:12:06.644565 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
I20240503 08:12:06.648384 26685 raft_consensus.cc:2932] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:06.649047 25960 raft_consensus.cc:2932] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
W20240503 08:12:06.656976 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
W20240503 08:12:06.664022 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39522: Illegal state: replica 56fb8e40a4da4b3794f07ad8137ecf42 is not leader of this config: current role FOLLOWER
W20240503 08:12:06.666503 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
I20240503 08:12:06.674981 24240 catalog_manager.cc:5484] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e reported cstate change: config changed from index 3 to 4, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a0d221e07029452cae34b8920884507e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:12:06.683240 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
W20240503 08:12:06.695245 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
I20240503 08:12:06.702822 24516 raft_consensus.cc:2932] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
W20240503 08:12:06.708619 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
W20240503 08:12:06.723824 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
W20240503 08:12:06.738152 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
W20240503 08:12:06.755970 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
I20240503 08:12:06.768106 26092 consensus_queue.cc:237] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
I20240503 08:12:06.774732 24646 raft_consensus.cc:1261] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Refusing update from remote peer f9705fdde3854639ba85097a161f37e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 2 index: 112. (index mismatch)
W20240503 08:12:06.776011 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
I20240503 08:12:06.777624 26671 consensus_queue.cc:1035] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240503 08:12:06.780320 25959 raft_consensus.cc:1261] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Refusing update from remote peer a0d221e07029452cae34b8920884507e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:12:06.781095 26680 consensus_queue.cc:1035] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:06.788836 26640 mvcc.cc:205] Tried to move back new op lower bound from 7023509204059824128 to 7023509202654937088. Current Snapshot: MvccSnapshot[applied={T|T < 7023509204059824128 or (T in {7023509204059824128})}]
I20240503 08:12:06.794389 24516 raft_consensus.cc:1261] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Refusing update from remote peer f9705fdde3854639ba85097a161f37e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 2 index: 112. (index mismatch)
I20240503 08:12:06.795292 26609 consensus_queue.cc:1035] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240503 08:12:06.799640 26685 raft_consensus.cc:2932] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
W20240503 08:12:06.801483 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
I20240503 08:12:06.800837 25960 raft_consensus.cc:2932] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:06.806634 24231 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6d422797de9843edbc473fca1d8e9e87 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:06.813972 24240 catalog_manager.cc:5484] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e reported cstate change: config changed from index 4 to 5, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New cstate: current_term: 2 leader_uuid: "a0d221e07029452cae34b8920884507e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:06.814107 25829 consensus_queue.cc:237] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 2.112, Last appended by leader: 110, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } }
I20240503 08:12:06.819985 24496 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d422797de9843edbc473fca1d8e9e87 with delete_type TABLET_DATA_TOMBSTONED (TS 6a640901590545eeb69e216d2399ea51 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42320
W20240503 08:12:06.822583 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
I20240503 08:12:06.825716 24646 raft_consensus.cc:1261] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Refusing update from remote peer f9705fdde3854639ba85097a161f37e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 112. Preceding OpId from leader: term: 2 index: 114. (index mismatch)
I20240503 08:12:06.827620 26609 consensus_queue.cc:1035] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20240503 08:12:06.844343 26697 tablet_replica.cc:330] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51: stopping tablet replica
I20240503 08:12:06.844591 26697 raft_consensus.cc:2229] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:06.845021 26697 raft_consensus.cc:2258] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:06.846032 26697 ts_tablet_manager.cc:1875] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:06.846406 26671 raft_consensus.cc:2932] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 [term 2 LEADER]: Committing config change with OpId 2.113: config changed from index 83 to 113, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
W20240503 08:12:06.849057 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
I20240503 08:12:06.851842 24229 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e0756a24eb5d4e26b2d44e46638a74be with cas_config_opid_index 83: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:06.855759 26697 ts_tablet_manager.cc:1888] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 08:12:06.855494 24646 raft_consensus.cc:2932] T e0756a24eb5d4e26b2d44e46638a74be P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Committing config change with OpId 2.113: config changed from index 83 to 113, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:06.859526 24240 catalog_manager.cc:5484] T e0756a24eb5d4e26b2d44e46638a74be P f9705fdde3854639ba85097a161f37e7 reported cstate change: config changed from index 83 to 113, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New cstate: current_term: 2 leader_uuid: "f9705fdde3854639ba85097a161f37e7" committed_config { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:06.871709 24496 tablet_service.cc:1514] Processing DeleteTablet for tablet e0756a24eb5d4e26b2d44e46638a74be with delete_type TABLET_DATA_TOMBSTONED (TS 6a640901590545eeb69e216d2399ea51 not found in new config with opid_index 113) from {username='slave'} at 127.0.0.1:42320
I20240503 08:12:06.872570 26697 log.cc:1196] T 6d422797de9843edbc473fca1d8e9e87 P 6a640901590545eeb69e216d2399ea51: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/6d422797de9843edbc473fca1d8e9e87
W20240503 08:12:06.873529 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
I20240503 08:12:06.874128 26697 tablet_replica.cc:330] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51: stopping tablet replica
I20240503 08:12:06.874758 26697 raft_consensus.cc:2229] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:06.874814 24230 catalog_manager.cc:4830] TS 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793): tablet 6d422797de9843edbc473fca1d8e9e87 (table TestTable [id=072979b49b8b4071923d58835e04c91c]) successfully deleted
I20240503 08:12:06.875324 26697 raft_consensus.cc:2258] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:06.890563 26697 ts_tablet_manager.cc:1875] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 08:12:06.898841 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
W20240503 08:12:06.925803 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
I20240503 08:12:06.944859 26697 ts_tablet_manager.cc:1888] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.112
I20240503 08:12:06.945900 26697 log.cc:1196] T e0756a24eb5d4e26b2d44e46638a74be P 6a640901590545eeb69e216d2399ea51: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/e0756a24eb5d4e26b2d44e46638a74be
I20240503 08:12:06.947242 24230 catalog_manager.cc:4830] TS 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793): tablet e0756a24eb5d4e26b2d44e46638a74be (table rebalance_test_table_3 [id=e6ccc802e346493aa6739ba0f901f14d]) successfully deleted
W20240503 08:12:06.950711 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
W20240503 08:12:06.977011 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
W20240503 08:12:07.002696 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
W20240503 08:12:07.031495 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
I20240503 08:12:07.062014 26401 raft_consensus.cc:979] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42: : Instructing follower b53502e2a1ae4c86a054d38048dd0df6 to start an election
I20240503 08:12:07.062218 26401 raft_consensus.cc:1067] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Signalling peer b53502e2a1ae4c86a054d38048dd0df6 to start an election
I20240503 08:12:07.063253 24386 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08df2ee2ca344cd2abd5ecb67ba9ffc1"
dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
from {username='slave'} at 127.20.107.131:45397
I20240503 08:12:07.063530 24386 raft_consensus.cc:482] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:07.063650 24386 raft_consensus.cc:3037] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:12:07.067335 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Service unavailable: leader transfer in progress
I20240503 08:12:07.084828 24386 raft_consensus.cc:504] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
I20240503 08:12:07.087355 24386 leader_election.cc:288] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 2 election: Requested vote from peers 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003), 9d313001b0a647caba8f53540a2b67ac (127.20.107.134:46393)
I20240503 08:12:07.090128 24646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08df2ee2ca344cd2abd5ecb67ba9ffc1" candidate_uuid: "b53502e2a1ae4c86a054d38048dd0df6" candidate_term: 2 candidate_status { last_received { term: 1 index: 152 } } ignore_live_leader: true dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42"
I20240503 08:12:07.090380 24646 raft_consensus.cc:3032] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:07.090497 24646 raft_consensus.cc:729] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Running, Role: LEADER
I20240503 08:12:07.090816 24646 consensus_queue.cc:260] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 152, Committed index: 152, Last appended: 1.152, Last appended by leader: 152, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
I20240503 08:12:07.091239 24646 raft_consensus.cc:3037] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:12:07.097358 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43298: Illegal state: replica 56fb8e40a4da4b3794f07ad8137ecf42 is not leader of this config: current role FOLLOWER
I20240503 08:12:07.099424 24646 raft_consensus.cc:2454] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b53502e2a1ae4c86a054d38048dd0df6 in term 2.
I20240503 08:12:07.105438 24320 leader_election.cc:302] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:12:07.105896 26617 raft_consensus.cc:2781] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:07.106037 26617 raft_consensus.cc:686] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 LEADER]: Becoming Leader. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Running, Role: LEADER
I20240503 08:12:07.106343 26617 consensus_queue.cc:237] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 152, Committed index: 152, Last appended: 1.152, Last appended by leader: 152, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
I20240503 08:12:07.110137 24240 catalog_manager.cc:5484] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: term changed from 1 to 2, leader changed from 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) to b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129). New cstate: current_term: 2 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:07.136809 24646 raft_consensus.cc:1261] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 152. Preceding OpId from leader: term: 2 index: 154. (index mismatch)
I20240503 08:12:07.137780 26642 consensus_queue.cc:1035] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 153, Last known committed idx: 152, Time since last communication: 0.000s
I20240503 08:12:07.142800 26395 mvcc.cc:205] Tried to move back new op lower bound from 7023509205545996288 to 7023509205430784000. Current Snapshot: MvccSnapshot[applied={T|T < 7023509205545996288 or (T in {7023509205545996288})}]
I20240503 08:12:07.238384 25960 raft_consensus.cc:3037] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:07.238335 25959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08df2ee2ca344cd2abd5ecb67ba9ffc1" candidate_uuid: "b53502e2a1ae4c86a054d38048dd0df6" candidate_term: 2 candidate_status { last_received { term: 1 index: 152 } } ignore_live_leader: true dest_uuid: "9d313001b0a647caba8f53540a2b67ac"
I20240503 08:12:07.243146 25960 raft_consensus.cc:1261] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 152. Preceding OpId from leader: term: 2 index: 154. (index mismatch)
I20240503 08:12:07.253978 26617 consensus_queue.cc:1035] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 153, Last known committed idx: 152, Time since last communication: 0.000s
I20240503 08:12:07.304061 24384 consensus_queue.cc:237] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 155, Committed index: 155, Last appended: 2.155, Last appended by leader: 152, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
W20240503 08:12:07.364046 24320 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.134:46393, user_credentials={real_user=slave}} blocked reactor thread for 67171us
I20240503 08:12:07.368201 25961 raft_consensus.cc:1261] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 2 index: 155. Preceding OpId from leader: term: 2 index: 156. (index mismatch)
W20240503 08:12:07.446518 24320 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.134:46393, user_credentials={real_user=slave}} blocked reactor thread for 51998.5us
I20240503 08:12:07.449244 26642 consensus_queue.cc:1035] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s
I20240503 08:12:07.453639 26641 raft_consensus.cc:2932] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 LEADER]: Committing config change with OpId 2.156: config changed from index 102 to 156, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:07.454311 25961 raft_consensus.cc:2932] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Committing config change with OpId 2.156: config changed from index 102 to 156, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:07.457932 24229 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1 with cas_config_opid_index 102: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:07.460055 24240 catalog_manager.cc:5484] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: config changed from index 102 to 156, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New cstate: current_term: 2 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:07.465915 24627 tablet_service.cc:1514] Processing DeleteTablet for tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1 with delete_type TABLET_DATA_TOMBSTONED (TS 56fb8e40a4da4b3794f07ad8137ecf42 not found in new config with opid_index 156) from {username='slave'} at 127.0.0.1:46200
I20240503 08:12:07.469919 26731 tablet_replica.cc:330] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42: stopping tablet replica
I20240503 08:12:07.471341 26731 raft_consensus.cc:2229] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:07.489161 26731 raft_consensus.cc:2258] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:07.490497 26731 ts_tablet_manager.cc:1875] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:07.511148 26731 ts_tablet_manager.cc:1888] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.155
I20240503 08:12:07.511499 26731 log.cc:1196] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 56fb8e40a4da4b3794f07ad8137ecf42: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/08df2ee2ca344cd2abd5ecb67ba9ffc1
I20240503 08:12:07.512456 24230 catalog_manager.cc:4830] TS 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003): tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1 (table rebalance_test_table_2 [id=4f8b8cf1869a436f9c012044eb10dc54]) successfully deleted
I20240503 08:12:08.029834 24646 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5dd5613d173f4b1eb0eacf2fec3a3ff0"
dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39558
I20240503 08:12:08.030105 24646 raft_consensus.cc:595] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Received request to transfer leadership
W20240503 08:12:08.090981 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.099689 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.126701 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.140236 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.170125 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.178936 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.202594 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.228317 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.247175 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.265997 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.279842 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.296553 24607 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.313727 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.332409 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.353232 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.379285 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
I20240503 08:12:08.387455 24517 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3c65da71faf34351885b5033d3d68835"
dest_uuid: "6a640901590545eeb69e216d2399ea51"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44700
I20240503 08:12:08.387694 24517 raft_consensus.cc:595] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:08.392378 26182 raft_consensus.cc:979] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51: : Instructing follower b53502e2a1ae4c86a054d38048dd0df6 to start an election
I20240503 08:12:08.392573 26237 raft_consensus.cc:1067] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Signalling peer b53502e2a1ae4c86a054d38048dd0df6 to start an election
I20240503 08:12:08.395367 24386 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c65da71faf34351885b5033d3d68835"
dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
from {username='slave'} at 127.20.107.130:39225
I20240503 08:12:08.395586 24386 raft_consensus.cc:482] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:08.395725 24386 raft_consensus.cc:3037] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:12:08.399958 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
I20240503 08:12:08.404654 24386 raft_consensus.cc:504] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
I20240503 08:12:08.406427 24386 leader_election.cc:288] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 2 election: Requested vote from peers 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793), 9d313001b0a647caba8f53540a2b67ac (127.20.107.134:46393)
I20240503 08:12:08.407624 25961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c65da71faf34351885b5033d3d68835" candidate_uuid: "b53502e2a1ae4c86a054d38048dd0df6" candidate_term: 2 candidate_status { last_received { term: 1 index: 140 } } ignore_live_leader: true dest_uuid: "9d313001b0a647caba8f53540a2b67ac"
I20240503 08:12:08.407897 25961 raft_consensus.cc:3037] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:08.409088 24517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c65da71faf34351885b5033d3d68835" candidate_uuid: "b53502e2a1ae4c86a054d38048dd0df6" candidate_term: 2 candidate_status { last_received { term: 1 index: 140 } } ignore_live_leader: true dest_uuid: "6a640901590545eeb69e216d2399ea51"
I20240503 08:12:08.409327 24517 raft_consensus.cc:3032] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:08.409440 24517 raft_consensus.cc:729] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Running, Role: LEADER
I20240503 08:12:08.409767 24517 consensus_queue.cc:260] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 140, Committed index: 140, Last appended: 1.140, Last appended by leader: 140, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
I20240503 08:12:08.410254 24517 raft_consensus.cc:3037] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:12:08.423974 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
I20240503 08:12:08.424849 25961 raft_consensus.cc:2454] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b53502e2a1ae4c86a054d38048dd0df6 in term 2.
I20240503 08:12:08.428890 24517 raft_consensus.cc:2454] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b53502e2a1ae4c86a054d38048dd0df6 in term 2.
I20240503 08:12:08.429462 24320 leader_election.cc:302] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:12:08.430531 26617 raft_consensus.cc:2781] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:08.430675 26617 raft_consensus.cc:686] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 LEADER]: Becoming Leader. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Running, Role: LEADER
I20240503 08:12:08.430972 26617 consensus_queue.cc:237] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 140, Committed index: 140, Last appended: 1.140, Last appended by leader: 140, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
I20240503 08:12:08.434756 24244 catalog_manager.cc:5484] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: term changed from 1 to 2, leader changed from 6a640901590545eeb69e216d2399ea51 (127.20.107.130) to b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129). New cstate: current_term: 2 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:08.452734 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
W20240503 08:12:08.459672 24474 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45022: Illegal state: replica 6a640901590545eeb69e216d2399ea51 is not leader of this config: current role FOLLOWER
I20240503 08:12:08.494303 26628 raft_consensus.cc:979] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42: : Instructing follower e7416e7c03a74a24b8fe2508111770d7 to start an election
I20240503 08:12:08.494490 26628 raft_consensus.cc:1067] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Signalling peer e7416e7c03a74a24b8fe2508111770d7 to start an election
W20240503 08:12:08.495460 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Service unavailable: leader transfer in progress
I20240503 08:12:08.496625 25697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5dd5613d173f4b1eb0eacf2fec3a3ff0"
dest_uuid: "e7416e7c03a74a24b8fe2508111770d7"
from {username='slave'} at 127.20.107.131:42857
I20240503 08:12:08.496927 25697 raft_consensus.cc:482] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:08.497098 25697 raft_consensus.cc:3037] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:08.500301 25697 raft_consensus.cc:504] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } }
I20240503 08:12:08.506501 25697 leader_election.cc:288] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [CANDIDATE]: Term 2 election: Requested vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775), 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:12:08.507792 24646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd5613d173f4b1eb0eacf2fec3a3ff0" candidate_uuid: "e7416e7c03a74a24b8fe2508111770d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 139 } } ignore_live_leader: true dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42"
I20240503 08:12:08.507998 24646 raft_consensus.cc:3032] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:08.508111 24646 raft_consensus.cc:729] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Running, Role: LEADER
I20240503 08:12:08.508410 24646 consensus_queue.cc:260] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 139, Committed index: 139, Last appended: 1.139, Last appended by leader: 139, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } }
I20240503 08:12:08.508814 24646 raft_consensus.cc:3037] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:08.512315 24646 raft_consensus.cc:2454] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7416e7c03a74a24b8fe2508111770d7 in term 2.
I20240503 08:12:08.513231 25633 leader_election.cc:302] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56fb8e40a4da4b3794f07ad8137ecf42, e7416e7c03a74a24b8fe2508111770d7; no voters:
W20240503 08:12:08.560355 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Illegal state: replica 56fb8e40a4da4b3794f07ad8137ecf42 is not leader of this config: current role FOLLOWER
W20240503 08:12:08.596468 24345 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45398: Illegal state: replica b53502e2a1ae4c86a054d38048dd0df6 is not leader of this config: current role FOLLOWER
I20240503 08:12:08.600234 26787 raft_consensus.cc:2781] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [term 2 FOLLOWER]: Leader election won for term 2
W20240503 08:12:08.600116 25633 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.107.131:34003, user_credentials={real_user=slave}} blocked reactor thread for 87026.8us
I20240503 08:12:08.610354 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd5613d173f4b1eb0eacf2fec3a3ff0" candidate_uuid: "e7416e7c03a74a24b8fe2508111770d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 139 } } ignore_live_leader: true dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
I20240503 08:12:08.610622 24386 raft_consensus.cc:3037] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:08.613485 24386 raft_consensus.cc:2454] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7416e7c03a74a24b8fe2508111770d7 in term 2.
I20240503 08:12:08.622280 26787 raft_consensus.cc:686] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [term 2 LEADER]: Becoming Leader. State: Replica: e7416e7c03a74a24b8fe2508111770d7, State: Running, Role: LEADER
I20240503 08:12:08.623028 26787 consensus_queue.cc:237] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 139, Committed index: 139, Last appended: 1.139, Last appended by leader: 139, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } }
W20240503 08:12:08.629832 24606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43276: Illegal state: replica 56fb8e40a4da4b3794f07ad8137ecf42 is not leader of this config: current role FOLLOWER
I20240503 08:12:08.629987 24243 catalog_manager.cc:5484] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 reported cstate change: term changed from 1 to 2, leader changed from 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) to e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132). New cstate: current_term: 2 leader_uuid: "e7416e7c03a74a24b8fe2508111770d7" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:08.639740 25960 raft_consensus.cc:1261] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 140. Preceding OpId from leader: term: 2 index: 142. (index mismatch)
I20240503 08:12:08.640822 26777 consensus_queue.cc:1035] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 140, Time since last communication: 0.000s
I20240503 08:12:08.641826 24516 raft_consensus.cc:1261] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 140. Preceding OpId from leader: term: 2 index: 142. (index mismatch)
I20240503 08:12:08.642627 26777 consensus_queue.cc:1035] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 140, Time since last communication: 0.000s
I20240503 08:12:08.650647 26754 mvcc.cc:205] Tried to move back new op lower bound from 7023509211698462720 to 7023509210856140800. Current Snapshot: MvccSnapshot[applied={T|T < 7023509211698462720}]
I20240503 08:12:08.654263 24386 consensus_queue.cc:237] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 142, Committed index: 142, Last appended: 2.142, Last appended by leader: 140, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
I20240503 08:12:08.656824 25961 raft_consensus.cc:1261] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 2 index: 142. Preceding OpId from leader: term: 2 index: 143. (index mismatch)
W20240503 08:12:08.660218 24345 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45398: Illegal state: replica b53502e2a1ae4c86a054d38048dd0df6 is not leader of this config: current role FOLLOWER
I20240503 08:12:08.660931 26642 consensus_queue.cc:1035] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 142, Time since last communication: 0.000s
I20240503 08:12:08.664678 26642 raft_consensus.cc:2932] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 LEADER]: Committing config change with OpId 2.143: config changed from index 104 to 143, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:08.669266 24229 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3c65da71faf34351885b5033d3d68835 with cas_config_opid_index 104: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:08.669799 25960 raft_consensus.cc:2932] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Committing config change with OpId 2.143: config changed from index 104 to 143, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } }
I20240503 08:12:08.671150 24244 catalog_manager.cc:5484] T 3c65da71faf34351885b5033d3d68835 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: config changed from index 104 to 143, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New cstate: current_term: 2 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:08.682431 24496 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c65da71faf34351885b5033d3d68835 with delete_type TABLET_DATA_TOMBSTONED (TS 6a640901590545eeb69e216d2399ea51 not found in new config with opid_index 143) from {username='slave'} at 127.0.0.1:42320
I20240503 08:12:08.716238 26796 tablet_replica.cc:330] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51: stopping tablet replica
I20240503 08:12:08.716562 26796 raft_consensus.cc:2229] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:08.716907 26796 raft_consensus.cc:2258] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:08.718791 26796 ts_tablet_manager.cc:1875] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:08.729336 26796 ts_tablet_manager.cc:1888] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.142
I20240503 08:12:08.729640 26796 log.cc:1196] T 3c65da71faf34351885b5033d3d68835 P 6a640901590545eeb69e216d2399ea51: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/3c65da71faf34351885b5033d3d68835
I20240503 08:12:08.730729 24230 catalog_manager.cc:4830] TS 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793): tablet 3c65da71faf34351885b5033d3d68835 (table rebalance_test_table_3 [id=e6ccc802e346493aa6739ba0f901f14d]) successfully deleted
I20240503 08:12:08.832712 24645 raft_consensus.cc:1261] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Refusing update from remote peer e7416e7c03a74a24b8fe2508111770d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 2 index: 141. (index mismatch)
I20240503 08:12:08.834934 26793 consensus_queue.cc:1035] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I20240503 08:12:08.838376 24386 raft_consensus.cc:1261] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Refusing update from remote peer e7416e7c03a74a24b8fe2508111770d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 2 index: 141. (index mismatch)
I20240503 08:12:08.840381 26793 consensus_queue.cc:1035] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I20240503 08:12:08.853677 26735 mvcc.cc:205] Tried to move back new op lower bound from 7023509212489793536 to 7023509211649396736. Current Snapshot: MvccSnapshot[applied={T|T < 7023509212489793536 or (T in {7023509212489793536})}]
I20240503 08:12:08.959358 25698 consensus_queue.cc:237] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 142, Committed index: 142, Last appended: 2.142, Last appended by leader: 139, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } }
I20240503 08:12:08.961822 24386 raft_consensus.cc:1261] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Refusing update from remote peer e7416e7c03a74a24b8fe2508111770d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 142. Preceding OpId from leader: term: 2 index: 143. (index mismatch)
I20240503 08:12:08.962399 26787 consensus_queue.cc:1035] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 142, Time since last communication: 0.000s
I20240503 08:12:08.974448 26793 raft_consensus.cc:2932] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [term 2 LEADER]: Committing config change with OpId 2.143: config changed from index 103 to 143, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:08.982767 24386 raft_consensus.cc:2932] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Committing config change with OpId 2.143: config changed from index 103 to 143, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:08.983157 24240 catalog_manager.cc:5484] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 reported cstate change: config changed from index 103 to 143, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New cstate: current_term: 2 leader_uuid: "e7416e7c03a74a24b8fe2508111770d7" committed_config { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:08.987763 24627 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0 with delete_type TABLET_DATA_TOMBSTONED (TS 56fb8e40a4da4b3794f07ad8137ecf42 not found in new config with opid_index 143) from {username='slave'} at 127.0.0.1:46200
I20240503 08:12:09.005064 24230 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0 with cas_config_opid_index 103: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:09.008970 26822 tablet_replica.cc:330] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42: stopping tablet replica
I20240503 08:12:09.009433 26822 raft_consensus.cc:2229] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:09.009963 26822 raft_consensus.cc:2258] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:09.011554 26822 ts_tablet_manager.cc:1875] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:09.032112 26822 ts_tablet_manager.cc:1888] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.142
I20240503 08:12:09.035146 26822 log.cc:1196] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P 56fb8e40a4da4b3794f07ad8137ecf42: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/5dd5613d173f4b1eb0eacf2fec3a3ff0
I20240503 08:12:09.045076 24230 catalog_manager.cc:4830] TS 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003): tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0 (table rebalance_test_table_0 [id=492713985fb14342af075b33cdc4b9af]) successfully deleted
I20240503 08:12:09.956822 24627 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:09.993026 24364 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:12:10.034392 26072 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:10.035169 24496 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:10.035355 25941 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:10.108737 25811 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:10.171252 25679 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:10.388267 24386 consensus_queue.cc:237] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 195, Committed index: 195, Last appended: 2.195, Last appended by leader: 152, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } }
W20240503 08:12:10.390444 24319 consensus_peers.cc:485] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 -> Peer f9705fdde3854639ba85097a161f37e7 (127.20.107.133:45569): Couldn't send request to peer f9705fdde3854639ba85097a161f37e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08df2ee2ca344cd2abd5ecb67ba9ffc1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:10.393101 25961 raft_consensus.cc:1261] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 2 index: 195. Preceding OpId from leader: term: 2 index: 196. (index mismatch)
I20240503 08:12:10.394987 26617 consensus_queue.cc:1035] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 196, Last known committed idx: 195, Time since last communication: 0.000s
I20240503 08:12:10.398077 26777 raft_consensus.cc:2932] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 LEADER]: Committing config change with OpId 2.196: config changed from index 156 to 196, NON_VOTER f9705fdde3854639ba85097a161f37e7 (127.20.107.133) added. New config: { opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } } }
I20240503 08:12:10.398670 25961 raft_consensus.cc:2932] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Committing config change with OpId 2.196: config changed from index 156 to 196, NON_VOTER f9705fdde3854639ba85097a161f37e7 (127.20.107.133) added. New config: { opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } } }
I20240503 08:12:10.404778 24240 catalog_manager.cc:5484] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: config changed from index 156 to 196, NON_VOTER f9705fdde3854639ba85097a161f37e7 (127.20.107.133) added. New cstate: current_term: 2 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:10.650714 24517 consensus_queue.cc:237] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 157, Committed index: 157, Last appended: 1.157, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } }
I20240503 08:12:10.653631 24383 raft_consensus.cc:1261] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
W20240503 08:12:10.653968 24451 consensus_peers.cc:485] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 -> Peer e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132:36475): Couldn't send request to peer e7416e7c03a74a24b8fe2508111770d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca9558ccb5444f61b3ee86625b2f1097. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:10.657673 26237 consensus_queue.cc:1035] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 08:12:10.660701 26237 raft_consensus.cc:2932] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.158: config changed from index -1 to 158, NON_VOTER e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) added. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } } }
I20240503 08:12:10.661773 24383 raft_consensus.cc:2932] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index -1 to 158, NON_VOTER e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) added. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } } }
I20240503 08:12:10.667011 24244 catalog_manager.cc:5484] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 reported cstate change: config changed from index -1 to 158, NON_VOTER e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) added. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:10.743243 25699 consensus_queue.cc:237] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 171, Committed index: 171, Last appended: 2.171, Last appended by leader: 139, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } }
I20240503 08:12:10.747306 24383 raft_consensus.cc:1261] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Refusing update from remote peer e7416e7c03a74a24b8fe2508111770d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 171. Preceding OpId from leader: term: 2 index: 172. (index mismatch)
I20240503 08:12:10.748518 26787 consensus_queue.cc:1035] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 172, Last known committed idx: 170, Time since last communication: 0.000s
I20240503 08:12:10.753798 26787 raft_consensus.cc:2932] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [term 2 LEADER]: Committing config change with OpId 2.172: config changed from index 143 to 172, NON_VOTER a0d221e07029452cae34b8920884507e (127.20.107.135) added. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } } }
I20240503 08:12:10.754285 24383 raft_consensus.cc:2932] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Committing config change with OpId 2.172: config changed from index 143 to 172, NON_VOTER a0d221e07029452cae34b8920884507e (127.20.107.135) added. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } } }
I20240503 08:12:10.759658 24244 catalog_manager.cc:5484] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: config changed from index 143 to 172, NON_VOTER a0d221e07029452cae34b8920884507e (127.20.107.135) added. New cstate: current_term: 2 leader_uuid: "e7416e7c03a74a24b8fe2508111770d7" committed_config { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } } }
W20240503 08:12:10.802016 25634 consensus_peers.cc:485] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 -> Peer a0d221e07029452cae34b8920884507e (127.20.107.135:45329): Couldn't send request to peer a0d221e07029452cae34b8920884507e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5dd5613d173f4b1eb0eacf2fec3a3ff0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:10.903321 26928 ts_tablet_manager.cc:902] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7: Initiating tablet copy from peer b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:12:10.938565 26928 tablet_copy_client.cc:308] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:46775
I20240503 08:12:10.939654 24406 tablet_copy_service.cc:140] P b53502e2a1ae4c86a054d38048dd0df6: Received BeginTabletCopySession request for tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1 from peer f9705fdde3854639ba85097a161f37e7 ({username='slave'} at 127.20.107.133:55825)
I20240503 08:12:10.939842 24406 tablet_copy_service.cc:161] P b53502e2a1ae4c86a054d38048dd0df6: Beginning new tablet copy session on tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1 from peer f9705fdde3854639ba85097a161f37e7 at {username='slave'} at 127.20.107.133:55825: session id = f9705fdde3854639ba85097a161f37e7-08df2ee2ca344cd2abd5ecb67ba9ffc1
I20240503 08:12:10.942991 24406 tablet_copy_source_session.cc:205] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:10.962772 26928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:10.971349 26928 tablet_copy_client.cc:783] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:10.971662 26928 tablet_copy_client.cc:647] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:10.974303 26928 tablet_copy_client.cc:523] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:10.979560 26928 tablet_bootstrap.cc:492] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7: Bootstrap starting.
I20240503 08:12:11.166883 26942 ts_tablet_manager.cc:902] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e: Initiating tablet copy from peer e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132:36475)
I20240503 08:12:11.223668 26942 tablet_copy_client.cc:308] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:36475
I20240503 08:12:11.268658 25719 tablet_copy_service.cc:140] P e7416e7c03a74a24b8fe2508111770d7: Received BeginTabletCopySession request for tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0 from peer a0d221e07029452cae34b8920884507e ({username='slave'} at 127.20.107.135:45127)
I20240503 08:12:11.268936 25719 tablet_copy_service.cc:161] P e7416e7c03a74a24b8fe2508111770d7: Beginning new tablet copy session on tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0 from peer a0d221e07029452cae34b8920884507e at {username='slave'} at 127.20.107.135:45127: session id = a0d221e07029452cae34b8920884507e-5dd5613d173f4b1eb0eacf2fec3a3ff0
I20240503 08:12:11.275370 25719 tablet_copy_source_session.cc:205] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:11.278999 26942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.286036 26952 ts_tablet_manager.cc:902] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7: Initiating tablet copy from peer 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:12:11.295830 26942 tablet_copy_client.cc:783] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:11.302906 26942 tablet_copy_client.cc:647] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:11.318974 26942 tablet_copy_client.cc:523] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:11.326155 26942 tablet_bootstrap.cc:492] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e: Bootstrap starting.
I20240503 08:12:11.426972 26952 tablet_copy_client.cc:308] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:43793
I20240503 08:12:11.427798 24537 tablet_copy_service.cc:140] P 6a640901590545eeb69e216d2399ea51: Received BeginTabletCopySession request for tablet ca9558ccb5444f61b3ee86625b2f1097 from peer e7416e7c03a74a24b8fe2508111770d7 ({username='slave'} at 127.20.107.132:44973)
I20240503 08:12:11.427999 24537 tablet_copy_service.cc:161] P 6a640901590545eeb69e216d2399ea51: Beginning new tablet copy session on tablet ca9558ccb5444f61b3ee86625b2f1097 from peer e7416e7c03a74a24b8fe2508111770d7 at {username='slave'} at 127.20.107.132:44973: session id = e7416e7c03a74a24b8fe2508111770d7-ca9558ccb5444f61b3ee86625b2f1097
I20240503 08:12:11.431010 24537 tablet_copy_source_session.cc:205] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:11.432421 26952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca9558ccb5444f61b3ee86625b2f1097. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.491505 26952 tablet_copy_client.cc:783] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:11.492031 26952 tablet_copy_client.cc:647] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:11.500036 26952 tablet_copy_client.cc:523] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:11.512931 26952 tablet_bootstrap.cc:492] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7: Bootstrap starting.
W20240503 08:12:11.971935 26942 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.068s sys 0.192s
W20240503 08:12:11.973129 26942 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.526s user 0.068s sys 0.192s
I20240503 08:12:12.011811 26952 tablet_bootstrap.cc:492] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=167 overwritten=0 applied=166 ignored=0} inserts{seen=838 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:12:12.012746 26952 tablet_bootstrap.cc:492] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7: Bootstrap complete.
I20240503 08:12:12.013161 26952 ts_tablet_manager.cc:1367] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7: Time spent bootstrapping tablet: real 0.500s user 0.225s sys 0.024s
I20240503 08:12:12.014699 26952 raft_consensus.cc:348] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } }
I20240503 08:12:12.016755 26952 raft_consensus.cc:729] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e7416e7c03a74a24b8fe2508111770d7, State: Initialized, Role: LEARNER
I20240503 08:12:12.017114 26952 consensus_queue.cc:260] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 166, Last appended: 1.167, Last appended by leader: 167, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: true } }
I20240503 08:12:12.017815 26952 ts_tablet_manager.cc:1398] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:12.020105 24537 tablet_copy_service.cc:342] P 6a640901590545eeb69e216d2399ea51: Request end of tablet copy session e7416e7c03a74a24b8fe2508111770d7-ca9558ccb5444f61b3ee86625b2f1097 received from {username='slave'} at 127.20.107.132:44973
I20240503 08:12:12.020238 24537 tablet_copy_service.cc:433] P 6a640901590545eeb69e216d2399ea51: ending tablet copy session e7416e7c03a74a24b8fe2508111770d7-ca9558ccb5444f61b3ee86625b2f1097 on tablet ca9558ccb5444f61b3ee86625b2f1097 with peer e7416e7c03a74a24b8fe2508111770d7
W20240503 08:12:12.049202 25765 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.131:34003, user_credentials={real_user=slave}} blocked reactor thread for 107771us
I20240503 08:12:12.186494 26928 tablet_bootstrap.cc:492] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=203 overwritten=0 applied=203 ignored=0} inserts{seen=830 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:12.187155 26928 tablet_bootstrap.cc:492] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7: Bootstrap complete.
I20240503 08:12:12.187738 26928 ts_tablet_manager.cc:1367] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7: Time spent bootstrapping tablet: real 1.208s user 0.287s sys 0.039s
I20240503 08:12:12.188613 26928 raft_consensus.cc:348] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } }
I20240503 08:12:12.190616 26928 raft_consensus.cc:729] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f9705fdde3854639ba85097a161f37e7, State: Initialized, Role: LEARNER
I20240503 08:12:12.190873 26928 consensus_queue.cc:260] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 203, Last appended: 2.203, Last appended by leader: 203, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: true } }
I20240503 08:12:12.191998 26928 ts_tablet_manager.cc:1398] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:12.194159 24406 tablet_copy_service.cc:342] P b53502e2a1ae4c86a054d38048dd0df6: Request end of tablet copy session f9705fdde3854639ba85097a161f37e7-08df2ee2ca344cd2abd5ecb67ba9ffc1 received from {username='slave'} at 127.20.107.133:55825
I20240503 08:12:12.194304 24406 tablet_copy_service.cc:433] P b53502e2a1ae4c86a054d38048dd0df6: ending tablet copy session f9705fdde3854639ba85097a161f37e7-08df2ee2ca344cd2abd5ecb67ba9ffc1 on tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1 with peer f9705fdde3854639ba85097a161f37e7
I20240503 08:12:12.328934 25831 raft_consensus.cc:1203] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.195->[2.196-2.220] Dedup: 2.203->[2.204-2.220]
I20240503 08:12:12.339419 25698 raft_consensus.cc:1203] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.157->[1.158-1.177] Dedup: 1.167->[1.168-1.177]
I20240503 08:12:12.440680 26641 raft_consensus.cc:1050] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6: attempting to promote NON_VOTER f9705fdde3854639ba85097a161f37e7 to VOTER
I20240503 08:12:12.441484 26641 consensus_queue.cc:237] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 222, Committed index: 222, Last appended: 2.222, Last appended by leader: 152, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } }
I20240503 08:12:12.443661 25960 raft_consensus.cc:1261] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 2 index: 222. Preceding OpId from leader: term: 2 index: 223. (index mismatch)
I20240503 08:12:12.444331 25830 raft_consensus.cc:1261] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [term 2 LEARNER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 2 index: 222. Preceding OpId from leader: term: 2 index: 223. (index mismatch)
I20240503 08:12:12.444830 26617 consensus_queue.cc:1035] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20240503 08:12:12.445364 26924 consensus_queue.cc:1035] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20240503 08:12:12.457602 26881 raft_consensus.cc:2932] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 LEADER]: Committing config change with OpId 2.223: config changed from index 196 to 223, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:12.459321 25830 raft_consensus.cc:2932] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [term 2 FOLLOWER]: Committing config change with OpId 2.223: config changed from index 196 to 223, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:12.465150 24244 catalog_manager.cc:5484] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: config changed from index 196 to 223, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:12.467542 25961 raft_consensus.cc:2932] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Committing config change with OpId 2.223: config changed from index 196 to 223, f9705fdde3854639ba85097a161f37e7 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:12.593573 26942 tablet_bootstrap.cc:492] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e: Bootstrap replayed 1/1 log segments. Stats: ops{read=183 overwritten=0 applied=183 ignored=0} inserts{seen=810 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:12.594146 26942 tablet_bootstrap.cc:492] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e: Bootstrap complete.
I20240503 08:12:12.594534 26942 ts_tablet_manager.cc:1367] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e: Time spent bootstrapping tablet: real 1.269s user 0.306s sys 0.214s
I20240503 08:12:12.595592 26942 raft_consensus.cc:348] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } }
I20240503 08:12:12.595888 26942 raft_consensus.cc:729] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a0d221e07029452cae34b8920884507e, State: Initialized, Role: LEARNER
I20240503 08:12:12.596109 26942 consensus_queue.cc:260] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 183, Last appended: 2.183, Last appended by leader: 183, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: true } }
I20240503 08:12:12.596856 26942 ts_tablet_manager.cc:1398] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:12.598229 25719 tablet_copy_service.cc:342] P e7416e7c03a74a24b8fe2508111770d7: Request end of tablet copy session a0d221e07029452cae34b8920884507e-5dd5613d173f4b1eb0eacf2fec3a3ff0 received from {username='slave'} at 127.20.107.135:45127
I20240503 08:12:12.598467 25719 tablet_copy_service.cc:433] P e7416e7c03a74a24b8fe2508111770d7: ending tablet copy session a0d221e07029452cae34b8920884507e-5dd5613d173f4b1eb0eacf2fec3a3ff0 on tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0 with peer a0d221e07029452cae34b8920884507e
I20240503 08:12:12.617293 26965 raft_consensus.cc:1050] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51: attempting to promote NON_VOTER e7416e7c03a74a24b8fe2508111770d7 to VOTER
I20240503 08:12:12.618117 26965 consensus_queue.cc:237] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 181, Committed index: 181, Last appended: 1.181, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } }
I20240503 08:12:12.620251 25698 raft_consensus.cc:1261] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7 [term 1 LEARNER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240503 08:12:12.621141 26965 consensus_queue.cc:1035] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240503 08:12:12.621265 24383 raft_consensus.cc:1261] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240503 08:12:12.621816 26856 consensus_queue.cc:1035] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240503 08:12:12.634075 26182 raft_consensus.cc:2932] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.182: config changed from index 158 to 182, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:12.640467 24383 raft_consensus.cc:2932] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 158 to 182, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:12.640658 24243 catalog_manager.cc:5484] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 reported cstate change: config changed from index 158 to 182, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:12.650475 25698 raft_consensus.cc:2932] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 158 to 182, e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:12.713543 24386 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08df2ee2ca344cd2abd5ecb67ba9ffc1"
dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45032
I20240503 08:12:12.713851 24386 raft_consensus.cc:595] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 LEADER]: Received request to transfer leadership
W20240503 08:12:12.726284 24344 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
W20240503 08:12:12.730741 24346 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
W20240503 08:12:12.739384 24346 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
W20240503 08:12:12.749325 24346 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
W20240503 08:12:12.755493 24346 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
W20240503 08:12:12.765586 24344 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
I20240503 08:12:12.767464 26093 raft_consensus.cc:1203] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.171->[2.172-2.209] Dedup: 2.183->[2.184-2.209]
W20240503 08:12:12.781467 24344 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
W20240503 08:12:12.793148 24346 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
W20240503 08:12:12.809886 24346 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
I20240503 08:12:12.836328 26917 raft_consensus.cc:1050] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7: attempting to promote NON_VOTER a0d221e07029452cae34b8920884507e to VOTER
I20240503 08:12:12.837189 26917 consensus_queue.cc:237] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 2.211, Last appended by leader: 139, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } }
I20240503 08:12:12.842244 24386 raft_consensus.cc:1261] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Refusing update from remote peer e7416e7c03a74a24b8fe2508111770d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 211. Preceding OpId from leader: term: 2 index: 212. (index mismatch)
I20240503 08:12:12.843474 26090 raft_consensus.cc:1261] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e [term 2 LEARNER]: Refusing update from remote peer e7416e7c03a74a24b8fe2508111770d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 211. Preceding OpId from leader: term: 2 index: 212. (index mismatch)
W20240503 08:12:12.843968 24344 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
I20240503 08:12:12.844056 26820 consensus_queue.cc:1035] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I20240503 08:12:12.847678 26820 raft_consensus.cc:2932] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [term 2 LEADER]: Committing config change with OpId 2.212: config changed from index 172 to 212, a0d221e07029452cae34b8920884507e (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:12.853453 26092 raft_consensus.cc:2932] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e [term 2 FOLLOWER]: Committing config change with OpId 2.212: config changed from index 172 to 212, a0d221e07029452cae34b8920884507e (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:12.854508 26917 consensus_queue.cc:1035] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I20240503 08:12:12.854736 24243 catalog_manager.cc:5484] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 reported cstate change: config changed from index 172 to 212, a0d221e07029452cae34b8920884507e (127.20.107.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e7416e7c03a74a24b8fe2508111770d7" committed_config { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:12.856768 24383 raft_consensus.cc:2932] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Committing config change with OpId 2.212: config changed from index 172 to 212, a0d221e07029452cae34b8920884507e (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:12.859752 25698 consensus_queue.cc:237] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 2.212, Last appended by leader: 139, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } }
I20240503 08:12:12.861482 26093 raft_consensus.cc:1261] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e [term 2 FOLLOWER]: Refusing update from remote peer e7416e7c03a74a24b8fe2508111770d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 212. Preceding OpId from leader: term: 2 index: 213. (index mismatch)
I20240503 08:12:12.862176 26917 consensus_queue.cc:1035] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I20240503 08:12:12.865371 26820 raft_consensus.cc:2932] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 [term 2 LEADER]: Committing config change with OpId 2.213: config changed from index 212 to 213, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
W20240503 08:12:12.866457 24344 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
I20240503 08:12:12.869297 24230 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0 with cas_config_opid_index 212: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:12.869639 26093 raft_consensus.cc:2932] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P a0d221e07029452cae34b8920884507e [term 2 FOLLOWER]: Committing config change with OpId 2.213: config changed from index 212 to 213, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } }
I20240503 08:12:12.871078 24243 catalog_manager.cc:5484] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P e7416e7c03a74a24b8fe2508111770d7 reported cstate change: config changed from index 212 to 213, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New cstate: current_term: 2 leader_uuid: "e7416e7c03a74a24b8fe2508111770d7" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:12.877087 24364 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0 with delete_type TABLET_DATA_TOMBSTONED (TS b53502e2a1ae4c86a054d38048dd0df6 not found in new config with opid_index 213) from {username='slave'} at 127.0.0.1:38614
W20240503 08:12:12.892663 24346 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
I20240503 08:12:12.910286 27023 tablet_replica.cc:330] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6: stopping tablet replica
I20240503 08:12:12.910506 27023 raft_consensus.cc:2229] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:12.910763 27023 raft_consensus.cc:2258] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:12.912616 27023 ts_tablet_manager.cc:1875] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 08:12:12.914417 24346 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
I20240503 08:12:12.922971 27023 ts_tablet_manager.cc:1888] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.212
I20240503 08:12:12.923255 27023 log.cc:1196] T 5dd5613d173f4b1eb0eacf2fec3a3ff0 P b53502e2a1ae4c86a054d38048dd0df6: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/5dd5613d173f4b1eb0eacf2fec3a3ff0
I20240503 08:12:12.924283 24229 catalog_manager.cc:4830] TS b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775): tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0 (table rebalance_test_table_0 [id=492713985fb14342af075b33cdc4b9af]) successfully deleted
W20240503 08:12:12.936478 24346 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
W20240503 08:12:12.955266 24346 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
W20240503 08:12:12.974129 24346 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
W20240503 08:12:12.992697 24344 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
W20240503 08:12:13.012833 24346 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
W20240503 08:12:13.035758 24346 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Service unavailable: leader transfer in progress
I20240503 08:12:13.039427 26617 raft_consensus.cc:979] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6: : Instructing follower f9705fdde3854639ba85097a161f37e7 to start an election
I20240503 08:12:13.039626 26617 raft_consensus.cc:1067] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 LEADER]: Signalling peer f9705fdde3854639ba85097a161f37e7 to start an election
I20240503 08:12:13.040416 25831 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08df2ee2ca344cd2abd5ecb67ba9ffc1"
dest_uuid: "f9705fdde3854639ba85097a161f37e7"
from {username='slave'} at 127.20.107.129:39715
I20240503 08:12:13.040622 25831 raft_consensus.cc:482] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:13.040714 25831 raft_consensus.cc:3037] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:13.043427 25831 raft_consensus.cc:504] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } }
I20240503 08:12:13.044656 24383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08df2ee2ca344cd2abd5ecb67ba9ffc1" candidate_uuid: "f9705fdde3854639ba85097a161f37e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 226 } } ignore_live_leader: true dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
I20240503 08:12:13.044903 24383 raft_consensus.cc:3032] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:12:13.045002 24383 raft_consensus.cc:729] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Running, Role: LEADER
I20240503 08:12:13.045353 24383 consensus_queue.cc:260] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 226, Committed index: 226, Last appended: 2.226, Last appended by leader: 226, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } }
I20240503 08:12:13.045733 24383 raft_consensus.cc:3037] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:13.048456 24383 raft_consensus.cc:2454] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9705fdde3854639ba85097a161f37e7 in term 3.
I20240503 08:12:13.049762 25764 leader_election.cc:302] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b53502e2a1ae4c86a054d38048dd0df6, f9705fdde3854639ba85097a161f37e7; no voters:
I20240503 08:12:13.051838 26700 raft_consensus.cc:2781] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:12:13.052004 26700 raft_consensus.cc:686] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [term 3 LEADER]: Becoming Leader. State: Replica: f9705fdde3854639ba85097a161f37e7, State: Running, Role: LEADER
I20240503 08:12:13.052557 26700 consensus_queue.cc:237] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 226, Committed index: 226, Last appended: 2.226, Last appended by leader: 226, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } }
W20240503 08:12:13.058022 24346 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45408: Illegal state: replica b53502e2a1ae4c86a054d38048dd0df6 is not leader of this config: current role FOLLOWER
I20240503 08:12:13.062592 25831 leader_election.cc:288] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [CANDIDATE]: Term 3 election: Requested vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775), 9d313001b0a647caba8f53540a2b67ac (127.20.107.134:46393)
I20240503 08:12:13.069532 24243 catalog_manager.cc:5484] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 reported cstate change: term changed from 2 to 3, leader changed from b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) to f9705fdde3854639ba85097a161f37e7 (127.20.107.133). New cstate: current_term: 3 leader_uuid: "f9705fdde3854639ba85097a161f37e7" committed_config { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:13.178134 25960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08df2ee2ca344cd2abd5ecb67ba9ffc1" candidate_uuid: "f9705fdde3854639ba85097a161f37e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 226 } } ignore_live_leader: true dest_uuid: "9d313001b0a647caba8f53540a2b67ac"
I20240503 08:12:13.178436 25960 raft_consensus.cc:3037] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Advancing to term 3
W20240503 08:12:13.181289 26165 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.20.107.190:44243, user_credentials={real_user=slave}} blocked reactor thread for 70038.1us
I20240503 08:12:13.191274 25960 raft_consensus.cc:2454] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9705fdde3854639ba85097a161f37e7 in term 3.
I20240503 08:12:13.197584 24516 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ca9558ccb5444f61b3ee86625b2f1097"
dest_uuid: "6a640901590545eeb69e216d2399ea51"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41096
I20240503 08:12:13.197850 24516 raft_consensus.cc:595] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Received request to transfer leadership
W20240503 08:12:13.224334 24476 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
I20240503 08:12:13.253158 24383 raft_consensus.cc:1261] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 3 FOLLOWER]: Refusing update from remote peer f9705fdde3854639ba85097a161f37e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 226. Preceding OpId from leader: term: 3 index: 228. (index mismatch)
W20240503 08:12:13.253154 24476 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
I20240503 08:12:13.257287 26700 consensus_queue.cc:1035] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 227, Last known committed idx: 226, Time since last communication: 0.000s
W20240503 08:12:13.259836 24476 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
W20240503 08:12:13.265683 24476 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
W20240503 08:12:13.276198 24476 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
I20240503 08:12:13.279270 25961 raft_consensus.cc:1261] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 3 FOLLOWER]: Refusing update from remote peer f9705fdde3854639ba85097a161f37e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 226. Preceding OpId from leader: term: 3 index: 228. (index mismatch)
I20240503 08:12:13.279942 26700 consensus_queue.cc:1035] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 227, Last known committed idx: 226, Time since last communication: 0.000s
W20240503 08:12:13.289230 24476 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
W20240503 08:12:13.299455 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
W20240503 08:12:13.308789 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
I20240503 08:12:13.335923 25830 consensus_queue.cc:237] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 3.230, Last appended by leader: 226, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } }
W20240503 08:12:13.336380 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
W20240503 08:12:13.351414 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
I20240503 08:12:13.360764 25959 raft_consensus.cc:1261] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 3 FOLLOWER]: Refusing update from remote peer f9705fdde3854639ba85097a161f37e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 230. Preceding OpId from leader: term: 3 index: 231. (index mismatch)
I20240503 08:12:13.361613 26700 consensus_queue.cc:1035] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 231, Last known committed idx: 230, Time since last communication: 0.000s
W20240503 08:12:13.368688 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
I20240503 08:12:13.376775 26700 raft_consensus.cc:2932] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P f9705fdde3854639ba85097a161f37e7 [term 3 LEADER]: Committing config change with OpId 3.231: config changed from index 223 to 231, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
W20240503 08:12:13.388002 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
I20240503 08:12:13.402293 25959 raft_consensus.cc:2932] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac [term 3 FOLLOWER]: Committing config change with OpId 3.231: config changed from index 223 to 231, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
W20240503 08:12:13.408517 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
I20240503 08:12:13.415125 24243 catalog_manager.cc:5484] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P 9d313001b0a647caba8f53540a2b67ac reported cstate change: config changed from index 223 to 231, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New cstate: current_term: 3 leader_uuid: "f9705fdde3854639ba85097a161f37e7" committed_config { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } } }
I20240503 08:12:13.420904 24364 tablet_service.cc:1514] Processing DeleteTablet for tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1 with delete_type TABLET_DATA_TOMBSTONED (TS b53502e2a1ae4c86a054d38048dd0df6 not found in new config with opid_index 231) from {username='slave'} at 127.0.0.1:38614
I20240503 08:12:13.421273 27023 tablet_replica.cc:330] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6: stopping tablet replica
I20240503 08:12:13.421491 27023 raft_consensus.cc:2229] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:13.421756 27023 raft_consensus.cc:2258] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:13.423988 27023 ts_tablet_manager.cc:1875] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 08:12:13.429404 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
I20240503 08:12:13.430790 27023 ts_tablet_manager.cc:1888] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.230
I20240503 08:12:13.430982 27023 log.cc:1196] T 08df2ee2ca344cd2abd5ecb67ba9ffc1 P b53502e2a1ae4c86a054d38048dd0df6: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/08df2ee2ca344cd2abd5ecb67ba9ffc1
I20240503 08:12:13.432108 24229 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1 with cas_config_opid_index 223: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:13.434259 24229 catalog_manager.cc:4830] TS b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775): tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1 (table rebalance_test_table_2 [id=4f8b8cf1869a436f9c012044eb10dc54]) successfully deleted
W20240503 08:12:13.446406 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
W20240503 08:12:13.473412 24476 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
W20240503 08:12:13.496356 24476 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
W20240503 08:12:13.516628 24476 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
W20240503 08:12:13.540768 24476 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
W20240503 08:12:13.562803 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
I20240503 08:12:13.582343 24516 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ca9558ccb5444f61b3ee86625b2f1097"
dest_uuid: "6a640901590545eeb69e216d2399ea51"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41110
I20240503 08:12:13.582577 24516 raft_consensus.cc:595] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Received request to transfer leadership
W20240503 08:12:13.589258 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
W20240503 08:12:13.614313 24475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Service unavailable: leader transfer in progress
I20240503 08:12:13.628073 26964 raft_consensus.cc:979] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51: : Instructing follower b53502e2a1ae4c86a054d38048dd0df6 to start an election
I20240503 08:12:13.628262 26964 raft_consensus.cc:1067] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Signalling peer b53502e2a1ae4c86a054d38048dd0df6 to start an election
I20240503 08:12:13.629671 24383 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ca9558ccb5444f61b3ee86625b2f1097"
dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
from {username='slave'} at 127.20.107.130:39225
I20240503 08:12:13.629902 24383 raft_consensus.cc:482] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:13.630036 24383 raft_consensus.cc:3037] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:13.633798 24383 raft_consensus.cc:504] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } }
I20240503 08:12:13.634601 24383 leader_election.cc:288] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 2 election: Requested vote from peers 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793), e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132:36475)
I20240503 08:12:13.637676 24516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9558ccb5444f61b3ee86625b2f1097" candidate_uuid: "b53502e2a1ae4c86a054d38048dd0df6" candidate_term: 2 candidate_status { last_received { term: 1 index: 194 } } ignore_live_leader: true dest_uuid: "6a640901590545eeb69e216d2399ea51"
I20240503 08:12:13.637918 24516 raft_consensus.cc:3032] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:13.638043 24516 raft_consensus.cc:729] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Running, Role: LEADER
I20240503 08:12:13.638326 24516 consensus_queue.cc:260] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 194, Committed index: 194, Last appended: 1.194, Last appended by leader: 194, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } }
I20240503 08:12:13.638711 24516 raft_consensus.cc:3037] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:12:13.642113 24476 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45018: Illegal state: replica 6a640901590545eeb69e216d2399ea51 is not leader of this config: current role FOLLOWER
I20240503 08:12:13.643005 24516 raft_consensus.cc:2454] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b53502e2a1ae4c86a054d38048dd0df6 in term 2.
I20240503 08:12:13.649956 24320 leader_election.cc:302] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, b53502e2a1ae4c86a054d38048dd0df6; no voters:
I20240503 08:12:13.650321 26881 raft_consensus.cc:2781] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:13.650488 26881 raft_consensus.cc:686] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 LEADER]: Becoming Leader. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Running, Role: LEADER
I20240503 08:12:13.650785 26881 consensus_queue.cc:237] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 194, Committed index: 194, Last appended: 1.194, Last appended by leader: 194, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } }
I20240503 08:12:13.655157 24243 catalog_manager.cc:5484] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: term changed from 1 to 2, leader changed from 6a640901590545eeb69e216d2399ea51 (127.20.107.130) to b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129). New cstate: current_term: 2 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:13.708101 25698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9558ccb5444f61b3ee86625b2f1097" candidate_uuid: "b53502e2a1ae4c86a054d38048dd0df6" candidate_term: 2 candidate_status { last_received { term: 1 index: 194 } } ignore_live_leader: true dest_uuid: "e7416e7c03a74a24b8fe2508111770d7"
I20240503 08:12:13.708462 25698 raft_consensus.cc:3037] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:13.711508 25698 raft_consensus.cc:2454] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b53502e2a1ae4c86a054d38048dd0df6 in term 2.
I20240503 08:12:13.737003 25697 raft_consensus.cc:1261] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7 [term 2 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 2 index: 196. (index mismatch)
I20240503 08:12:13.737754 24516 raft_consensus.cc:1261] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 2 index: 196. (index mismatch)
I20240503 08:12:13.740046 26641 consensus_queue.cc:1035] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 194, Time since last communication: 0.000s
I20240503 08:12:13.740388 26924 consensus_queue.cc:1035] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 194, Time since last communication: 0.000s
I20240503 08:12:13.758242 24383 consensus_queue.cc:237] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 196, Committed index: 196, Last appended: 2.196, Last appended by leader: 194, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } }
I20240503 08:12:13.760208 25697 raft_consensus.cc:1261] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7 [term 2 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 2 index: 196. Preceding OpId from leader: term: 2 index: 197. (index mismatch)
I20240503 08:12:13.761304 26881 consensus_queue.cc:1035] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 197, Last known committed idx: 196, Time since last communication: 0.000s
I20240503 08:12:13.764073 26617 raft_consensus.cc:2932] T ca9558ccb5444f61b3ee86625b2f1097 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 LEADER]: Committing config change with OpId 2.197: config changed from index 182 to 197, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New config: { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:13.764681 25697 raft_consensus.cc:2932] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7 [term 2 FOLLOWER]: Committing config change with OpId 2.197: config changed from index 182 to 197, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New config: { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:13.783347 24240 catalog_manager.cc:5484] T ca9558ccb5444f61b3ee86625b2f1097 P e7416e7c03a74a24b8fe2508111770d7 reported cstate change: config changed from index 182 to 197, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New cstate: current_term: 2 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } }
I20240503 08:12:13.783948 24229 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ca9558ccb5444f61b3ee86625b2f1097 with cas_config_opid_index 182: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:13.789340 24496 tablet_service.cc:1514] Processing DeleteTablet for tablet ca9558ccb5444f61b3ee86625b2f1097 with delete_type TABLET_DATA_TOMBSTONED (TS 6a640901590545eeb69e216d2399ea51 not found in new config with opid_index 197) from {username='slave'} at 127.0.0.1:42320
I20240503 08:12:13.804409 27089 tablet_replica.cc:330] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51: stopping tablet replica
I20240503 08:12:13.804790 27089 raft_consensus.cc:2229] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:13.805156 27089 raft_consensus.cc:2258] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:13.806358 27089 ts_tablet_manager.cc:1875] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:13.815068 27089 ts_tablet_manager.cc:1888] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.196
I20240503 08:12:13.815313 27089 log.cc:1196] T ca9558ccb5444f61b3ee86625b2f1097 P 6a640901590545eeb69e216d2399ea51: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/ca9558ccb5444f61b3ee86625b2f1097
I20240503 08:12:13.817752 24230 catalog_manager.cc:4830] TS 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793): tablet ca9558ccb5444f61b3ee86625b2f1097 (table rebalance_test_table_1 [id=bb0497226b9c4da186bef24f42f0762a]) successfully deleted
I20240503 08:12:14.394215 26072 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:14.410008 24364 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:12:14.420249 24496 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:14.422256 24627 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:14.432833 25941 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:14.438477 25811 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:14.579623 25679 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:15.712177 25941 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:15.734246 24496 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:15.736563 26072 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:15.789870 24364 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:12:15.799486 24627 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:15.845458 25679 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:15.907990 25811 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:12:16.614089 24582 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.135:45329, user_credentials={real_user=slave}} blocked reactor thread for 52394.1us
I20240503 08:12:17.988303 24496 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:18.010541 26072 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:18.019927 25679 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:18.029493 24364 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:12:18.045913 25811 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:18.061797 24627 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:18.106858 25941 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:19.143729 24627 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:19.187988 24496 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:19.251528 26072 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:19.259244 25941 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:19.294854 25679 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:19.365420 25811 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:19.380846 24364 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:12:20.453461 24364 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:12:20.456264 25811 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:20.460393 26072 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:20.481559 25941 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:20.503434 25679 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:20.520852 24627 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:20.522470 24496 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
4a981765e30a453eab8294095153bf23 | 127.20.107.190:44243 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.107.148:36581 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
56fb8e40a4da4b3794f07ad8137ecf42 | 127.20.107.131:34003 | HEALTHY | <none> | 4 | 0
6a640901590545eeb69e216d2399ea51 | 127.20.107.130:43793 | HEALTHY | <none> | 3 | 0
9d313001b0a647caba8f53540a2b67ac | 127.20.107.134:46393 | HEALTHY | <none> | 0 | 0
a0d221e07029452cae34b8920884507e | 127.20.107.135:45329 | HEALTHY | <none> | 1 | 0
b53502e2a1ae4c86a054d38048dd0df6 | 127.20.107.129:46775 | HEALTHY | <none> | 5 | 0
e7416e7c03a74a24b8fe2508111770d7 | 127.20.107.132:36475 | HEALTHY | <none> | 1 | 0
f9705fdde3854639ba85097a161f37e7 | 127.20.107.133:45569 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.20.107.129 | experimental | 127.20.107.129:46775
local_ip_for_outbound_sockets | 127.20.107.130 | experimental | 127.20.107.130:43793
local_ip_for_outbound_sockets | 127.20.107.131 | experimental | 127.20.107.131:34003
local_ip_for_outbound_sockets | 127.20.107.132 | experimental | 127.20.107.132:36475
local_ip_for_outbound_sockets | 127.20.107.133 | experimental | 127.20.107.133:45569
local_ip_for_outbound_sockets | 127.20.107.134 | experimental | 127.20.107.134:46393
local_ip_for_outbound_sockets | 127.20.107.135 | experimental | 127.20.107.135:45329
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.20.107.129:46775
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.20.107.130:43793
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.20.107.131:34003
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.20.107.132:36475
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.20.107.133:45569
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.20.107.134:46393
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.20.107.135:45329
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.107.148:36581 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:12:20.664283 20910 log_verifier.cc:126] Checking tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1
I20240503 08:12:20.789547 20910 log_verifier.cc:177] Verified matching terms for 349 ops in tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1
I20240503 08:12:20.790030 20910 log_verifier.cc:126] Checking tablet 3071edf0c1004a6496d1cb46591d422e
I20240503 08:12:20.927400 20910 log_verifier.cc:177] Verified matching terms for 348 ops in tablet 3071edf0c1004a6496d1cb46591d422e
I20240503 08:12:20.927907 20910 log_verifier.cc:126] Checking tablet 39ccbb3b60094ed1a8a09ae6b6c4e6a5
I20240503 08:12:21.086431 20910 log_verifier.cc:177] Verified matching terms for 379 ops in tablet 39ccbb3b60094ed1a8a09ae6b6c4e6a5
I20240503 08:12:21.086947 20910 log_verifier.cc:126] Checking tablet 3c65da71faf34351885b5033d3d68835
I20240503 08:12:21.230177 20910 log_verifier.cc:177] Verified matching terms for 347 ops in tablet 3c65da71faf34351885b5033d3d68835
I20240503 08:12:21.230623 20910 log_verifier.cc:126] Checking tablet 52edf3b518be4f45bb347dbbe4bf7fd9
I20240503 08:12:21.357170 20910 log_verifier.cc:177] Verified matching terms for 348 ops in tablet 52edf3b518be4f45bb347dbbe4bf7fd9
I20240503 08:12:21.357627 20910 log_verifier.cc:126] Checking tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0
I20240503 08:12:21.516536 20910 log_verifier.cc:177] Verified matching terms for 374 ops in tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0
I20240503 08:12:21.517068 20910 log_verifier.cc:126] Checking tablet 61bf5a9f26034821b821a45ac37ba133
I20240503 08:12:21.658267 20910 log_verifier.cc:177] Verified matching terms for 374 ops in tablet 61bf5a9f26034821b821a45ac37ba133
I20240503 08:12:21.658742 20910 log_verifier.cc:126] Checking tablet 6d422797de9843edbc473fca1d8e9e87
I20240503 08:12:21.676391 20910 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 6d422797de9843edbc473fca1d8e9e87
I20240503 08:12:21.676507 20910 log_verifier.cc:126] Checking tablet 7b858bcd6e1c469d9f19f142106908d3
I20240503 08:12:21.815694 20910 log_verifier.cc:177] Verified matching terms for 341 ops in tablet 7b858bcd6e1c469d9f19f142106908d3
I20240503 08:12:21.816154 20910 log_verifier.cc:126] Checking tablet 7d2772d08c06498db2cb0b02497ea042
I20240503 08:12:21.935292 20910 log_verifier.cc:177] Verified matching terms for 330 ops in tablet 7d2772d08c06498db2cb0b02497ea042
I20240503 08:12:21.935746 20910 log_verifier.cc:126] Checking tablet 83697ce9030f4981af717d18eb6757a4
I20240503 08:12:22.084852 20910 log_verifier.cc:177] Verified matching terms for 379 ops in tablet 83697ce9030f4981af717d18eb6757a4
I20240503 08:12:22.085497 20910 log_verifier.cc:126] Checking tablet c85cf905ebfa4e7b91a0af9ed2fa77da
I20240503 08:12:22.224826 20910 log_verifier.cc:177] Verified matching terms for 370 ops in tablet c85cf905ebfa4e7b91a0af9ed2fa77da
I20240503 08:12:22.225353 20910 log_verifier.cc:126] Checking tablet ca9558ccb5444f61b3ee86625b2f1097
I20240503 08:12:22.349871 20910 log_verifier.cc:177] Verified matching terms for 316 ops in tablet ca9558ccb5444f61b3ee86625b2f1097
I20240503 08:12:22.350298 20910 log_verifier.cc:126] Checking tablet d73b0d5398cd484ba45515cf95cd5b14
I20240503 08:12:22.504817 20910 log_verifier.cc:177] Verified matching terms for 377 ops in tablet d73b0d5398cd484ba45515cf95cd5b14
I20240503 08:12:22.505307 20910 log_verifier.cc:126] Checking tablet d8a32af7adaf410582e1b898257db62d
I20240503 08:12:22.648376 20910 log_verifier.cc:177] Verified matching terms for 397 ops in tablet d8a32af7adaf410582e1b898257db62d
I20240503 08:12:22.648895 20910 log_verifier.cc:126] Checking tablet e0756a24eb5d4e26b2d44e46638a74be
I20240503 08:12:22.792177 20910 log_verifier.cc:177] Verified matching terms for 352 ops in tablet e0756a24eb5d4e26b2d44e46638a74be
I20240503 08:12:22.795501 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.136:0
--local_ip_for_outbound_sockets=127.20.107.136
--webserver_interface=127.20.107.136
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--builtin_ntp_servers=127.20.107.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:12:23.172140 27398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:23.172442 27398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:23.172580 27398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:23.180778 27398 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:23.180971 27398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:23.181135 27398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.136
I20240503 08:12:23.190801 27398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.107.136
--webserver_port=0
--tserver_master_addrs=127.20.107.190:44243
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.136
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:23.192193 27398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:23.194033 27398 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:23.207669 27404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:23.208130 27405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:23.208964 27407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:23.210880 27398 server_base.cc:1034] running on GCE node
I20240503 08:12:23.211495 27398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:23.212443 27398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:23.213655 27398 hybrid_clock.cc:648] HybridClock initialized: now 1714723943213605 us; error 45 us; skew 500 ppm
I20240503 08:12:23.214002 27398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:23.220713 27398 webserver.cc:460] Webserver started at http://127.20.107.136:43761/ using document root <none> and password file <none>
I20240503 08:12:23.221522 27398 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:23.221643 27398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:23.221961 27398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:23.224821 27398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "efe9f2f995024b01b2ade8bdc404a2a3"
format_stamp: "Formatted at 2024-05-03 08:12:23 on dist-test-slave-4tl1"
I20240503 08:12:23.225656 27398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "efe9f2f995024b01b2ade8bdc404a2a3"
format_stamp: "Formatted at 2024-05-03 08:12:23 on dist-test-slave-4tl1"
I20240503 08:12:23.230887 27398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:12:23.234478 27413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:23.235805 27398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:23.235970 27398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "efe9f2f995024b01b2ade8bdc404a2a3"
format_stamp: "Formatted at 2024-05-03 08:12:23 on dist-test-slave-4tl1"
I20240503 08:12:23.236174 27398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:23.256227 27398 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:23.257352 27398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:23.257679 27398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:23.258837 27398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:23.260507 27398 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:23.260615 27398 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:23.260759 27398 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:23.260841 27398 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:23.301800 27398 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.136:41565
I20240503 08:12:23.301874 27525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.136:41565 every 8 connection(s)
I20240503 08:12:23.303113 27398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:12:23.308809 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 27398
I20240503 08:12:23.309363 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:12:23.319329 27526 heartbeater.cc:347] Connected to a master server at 127.20.107.190:44243
I20240503 08:12:23.319727 27526 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:23.320858 27526 heartbeater.cc:511] Master 127.20.107.190:44243 requested a full tablet report, sending...
I20240503 08:12:23.322470 24240 ts_manager.cc:194] Registered new tserver with Master: efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136:41565)
I20240503 08:12:23.323659 24240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.136:45279
I20240503 08:12:23.945367 24627 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:23.964639 26072 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:23.975019 24496 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:23.980135 24364 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:12:23.999984 25941 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:24.025839 25679 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:24.046146 27461 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:24.062530 25811 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:24.265666 24627 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:24.282248 24496 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:24.282881 24364 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:12:24.286381 26072 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:24.298621 25941 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:24.305207 27461 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:24.317409 25679 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:24.327008 27526 heartbeater.cc:503] Master 127.20.107.190:44243 was elected leader, sending a full tablet report...
I20240503 08:12:24.331806 25811 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:24.459760 24515 consensus_queue.cc:237] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 1.348, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } }
I20240503 08:12:24.462972 25697 raft_consensus.cc:1261] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 348. Preceding OpId from leader: term: 1 index: 349. (index mismatch)
I20240503 08:12:24.463722 26965 consensus_queue.cc:1035] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
W20240503 08:12:24.478755 24450 consensus_peers.cc:485] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 -> Peer efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136:41565): Couldn't send request to peer efe9f2f995024b01b2ade8bdc404a2a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3071edf0c1004a6496d1cb46591d422e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:24.481127 26965 raft_consensus.cc:2932] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.349: config changed from index 87 to 349, NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) added. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } } }
I20240503 08:12:24.487219 25697 raft_consensus.cc:2932] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Committing config change with OpId 1.349: config changed from index 87 to 349, NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) added. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } } }
I20240503 08:12:24.494975 24236 catalog_manager.cc:5484] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 reported cstate change: config changed from index 87 to 349, NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } } }
I20240503 08:12:24.522302 26090 consensus_queue.cc:237] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } }
I20240503 08:12:24.525264 25961 raft_consensus.cc:1261] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Refusing update from remote peer a0d221e07029452cae34b8920884507e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:12:24.525848 27531 consensus_queue.cc:1035] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:24.528960 27636 raft_consensus.cc:2932] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } } }
I20240503 08:12:24.529831 25961 raft_consensus.cc:2932] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } } }
W20240503 08:12:24.532723 26024 consensus_peers.cc:485] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e -> Peer efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136:41565): Couldn't send request to peer efe9f2f995024b01b2ade8bdc404a2a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d422797de9843edbc473fca1d8e9e87. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:24.534612 24240 catalog_manager.cc:5484] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e reported cstate change: config changed from index 5 to 6, NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) added. New cstate: current_term: 2 leader_uuid: "a0d221e07029452cae34b8920884507e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:24.546077 24383 consensus_queue.cc:237] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 1.348, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } }
I20240503 08:12:24.548491 24515 raft_consensus.cc:1261] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 348. Preceding OpId from leader: term: 1 index: 349. (index mismatch)
I20240503 08:12:24.549181 27615 consensus_queue.cc:1035] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20240503 08:12:24.552356 27615 raft_consensus.cc:2932] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 LEADER]: Committing config change with OpId 1.349: config changed from index -1 to 349, NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) added. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } } }
W20240503 08:12:24.556155 24319 consensus_peers.cc:485] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 -> Peer efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136:41565): Couldn't send request to peer efe9f2f995024b01b2ade8bdc404a2a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52edf3b518be4f45bb347dbbe4bf7fd9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:24.556771 24515 raft_consensus.cc:2932] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Committing config change with OpId 1.349: config changed from index -1 to 349, NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) added. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } } }
I20240503 08:12:24.559082 24236 catalog_manager.cc:5484] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 reported cstate change: config changed from index -1 to 349, NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:24.579810 24646 consensus_queue.cc:237] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 377, Committed index: 377, Last appended: 1.377, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } }
I20240503 08:12:24.584101 26090 raft_consensus.cc:1261] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 377. Preceding OpId from leader: term: 1 index: 378. (index mismatch)
I20240503 08:12:24.584906 27396 consensus_queue.cc:1035] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 378, Last known committed idx: 377, Time since last communication: 0.000s
I20240503 08:12:24.587999 27655 raft_consensus.cc:2932] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.378: config changed from index 102 to 378, NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) added. New config: { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } } }
I20240503 08:12:24.588611 26090 raft_consensus.cc:2932] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Committing config change with OpId 1.378: config changed from index 102 to 378, NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) added. New config: { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } } }
W20240503 08:12:24.589902 24580 consensus_peers.cc:485] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 -> Peer efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136:41565): Couldn't send request to peer efe9f2f995024b01b2ade8bdc404a2a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d73b0d5398cd484ba45515cf95cd5b14. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:24.593261 24236 catalog_manager.cc:5484] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e reported cstate change: config changed from index 102 to 378, NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } } }
I20240503 08:12:24.943441 27699 ts_tablet_manager.cc:902] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3: Initiating tablet copy from peer a0d221e07029452cae34b8920884507e (127.20.107.135:45329)
I20240503 08:12:24.945194 27699 tablet_copy_client.cc:308] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.135:45329
I20240503 08:12:24.952046 26113 tablet_copy_service.cc:140] P a0d221e07029452cae34b8920884507e: Received BeginTabletCopySession request for tablet 6d422797de9843edbc473fca1d8e9e87 from peer efe9f2f995024b01b2ade8bdc404a2a3 ({username='slave'} at 127.20.107.136:40225)
I20240503 08:12:24.952226 26113 tablet_copy_service.cc:161] P a0d221e07029452cae34b8920884507e: Beginning new tablet copy session on tablet 6d422797de9843edbc473fca1d8e9e87 from peer efe9f2f995024b01b2ade8bdc404a2a3 at {username='slave'} at 127.20.107.136:40225: session id = efe9f2f995024b01b2ade8bdc404a2a3-6d422797de9843edbc473fca1d8e9e87
I20240503 08:12:24.954281 26113 tablet_copy_source_session.cc:205] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:24.958369 27699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d422797de9843edbc473fca1d8e9e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.969102 27699 tablet_copy_client.cc:783] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:24.969529 27699 tablet_copy_client.cc:647] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:24.972326 27699 tablet_copy_client.cc:523] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:24.976091 27699 tablet_bootstrap.cc:492] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3: Bootstrap starting.
I20240503 08:12:24.983983 27699 log.cc:824] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:24.989153 27699 tablet_bootstrap.cc:492] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:24.989603 27699 tablet_bootstrap.cc:492] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3: Bootstrap complete.
I20240503 08:12:24.989995 27699 ts_tablet_manager.cc:1367] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3: Time spent bootstrapping tablet: real 0.014s user 0.015s sys 0.001s
I20240503 08:12:24.995921 27699 raft_consensus.cc:348] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } }
I20240503 08:12:24.996331 27699 raft_consensus.cc:729] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: efe9f2f995024b01b2ade8bdc404a2a3, State: Initialized, Role: LEARNER
I20240503 08:12:24.996805 27699 consensus_queue.cc:260] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } }
I20240503 08:12:24.998272 27699 ts_tablet_manager.cc:1398] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240503 08:12:24.999337 26113 tablet_copy_service.cc:342] P a0d221e07029452cae34b8920884507e: Request end of tablet copy session efe9f2f995024b01b2ade8bdc404a2a3-6d422797de9843edbc473fca1d8e9e87 received from {username='slave'} at 127.20.107.136:40225
I20240503 08:12:24.999469 26113 tablet_copy_service.cc:433] P a0d221e07029452cae34b8920884507e: ending tablet copy session efe9f2f995024b01b2ade8bdc404a2a3-6d422797de9843edbc473fca1d8e9e87 on tablet 6d422797de9843edbc473fca1d8e9e87 with peer efe9f2f995024b01b2ade8bdc404a2a3
I20240503 08:12:25.083565 27699 ts_tablet_manager.cc:902] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3: Initiating tablet copy from peer 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793)
I20240503 08:12:25.085286 27699 tablet_copy_client.cc:308] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:43793
I20240503 08:12:25.091610 24537 tablet_copy_service.cc:140] P 6a640901590545eeb69e216d2399ea51: Received BeginTabletCopySession request for tablet 3071edf0c1004a6496d1cb46591d422e from peer efe9f2f995024b01b2ade8bdc404a2a3 ({username='slave'} at 127.20.107.136:56121)
I20240503 08:12:25.091763 24537 tablet_copy_service.cc:161] P 6a640901590545eeb69e216d2399ea51: Beginning new tablet copy session on tablet 3071edf0c1004a6496d1cb46591d422e from peer efe9f2f995024b01b2ade8bdc404a2a3 at {username='slave'} at 127.20.107.136:56121: session id = efe9f2f995024b01b2ade8bdc404a2a3-3071edf0c1004a6496d1cb46591d422e
I20240503 08:12:25.093940 24537 tablet_copy_source_session.cc:205] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:25.095574 27699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3071edf0c1004a6496d1cb46591d422e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:25.101233 27699 tablet_copy_client.cc:783] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:25.101437 27699 tablet_copy_client.cc:647] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:25.104048 27699 tablet_copy_client.cc:523] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:25.107615 27699 tablet_bootstrap.cc:492] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3: Bootstrap starting.
I20240503 08:12:25.158989 27732 ts_tablet_manager.cc:902] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3: Initiating tablet copy from peer b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:12:25.165537 27731 ts_tablet_manager.cc:902] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3: Initiating tablet copy from peer 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003)
I20240503 08:12:25.184016 27731 tablet_copy_client.cc:308] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:34003
I20240503 08:12:25.184178 27732 tablet_copy_client.cc:308] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:46775
I20240503 08:12:25.191617 24667 tablet_copy_service.cc:140] P 56fb8e40a4da4b3794f07ad8137ecf42: Received BeginTabletCopySession request for tablet d73b0d5398cd484ba45515cf95cd5b14 from peer efe9f2f995024b01b2ade8bdc404a2a3 ({username='slave'} at 127.20.107.136:43169)
I20240503 08:12:25.191809 24667 tablet_copy_service.cc:161] P 56fb8e40a4da4b3794f07ad8137ecf42: Beginning new tablet copy session on tablet d73b0d5398cd484ba45515cf95cd5b14 from peer efe9f2f995024b01b2ade8bdc404a2a3 at {username='slave'} at 127.20.107.136:43169: session id = efe9f2f995024b01b2ade8bdc404a2a3-d73b0d5398cd484ba45515cf95cd5b14
I20240503 08:12:25.194195 24406 tablet_copy_service.cc:140] P b53502e2a1ae4c86a054d38048dd0df6: Received BeginTabletCopySession request for tablet 52edf3b518be4f45bb347dbbe4bf7fd9 from peer efe9f2f995024b01b2ade8bdc404a2a3 ({username='slave'} at 127.20.107.136:43581)
I20240503 08:12:25.194368 24406 tablet_copy_service.cc:161] P b53502e2a1ae4c86a054d38048dd0df6: Beginning new tablet copy session on tablet 52edf3b518be4f45bb347dbbe4bf7fd9 from peer efe9f2f995024b01b2ade8bdc404a2a3 at {username='slave'} at 127.20.107.136:43581: session id = efe9f2f995024b01b2ade8bdc404a2a3-52edf3b518be4f45bb347dbbe4bf7fd9
I20240503 08:12:25.194753 24667 tablet_copy_source_session.cc:205] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:25.196297 27731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73b0d5398cd484ba45515cf95cd5b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:25.197082 24406 tablet_copy_source_session.cc:205] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:25.198518 27732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52edf3b518be4f45bb347dbbe4bf7fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:25.203786 27731 tablet_copy_client.cc:783] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:25.204057 27731 tablet_copy_client.cc:647] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:25.205581 27732 tablet_copy_client.cc:783] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:25.205888 27732 tablet_copy_client.cc:647] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:25.207020 27731 tablet_copy_client.cc:523] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:25.209326 27732 tablet_copy_client.cc:523] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:25.210811 27731 tablet_bootstrap.cc:492] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3: Bootstrap starting.
I20240503 08:12:25.211927 27732 tablet_bootstrap.cc:492] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3: Bootstrap starting.
I20240503 08:12:25.558616 27481 raft_consensus.cc:1203] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
I20240503 08:12:25.631714 27699 tablet_bootstrap.cc:492] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=349 overwritten=0 applied=349 ignored=0} inserts{seen=1031 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:25.632414 27699 tablet_bootstrap.cc:492] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3: Bootstrap complete.
I20240503 08:12:25.632786 27699 ts_tablet_manager.cc:1367] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3: Time spent bootstrapping tablet: real 0.525s user 0.444s sys 0.029s
I20240503 08:12:25.633932 27699 raft_consensus.cc:348] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } }
I20240503 08:12:25.634233 27699 raft_consensus.cc:729] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efe9f2f995024b01b2ade8bdc404a2a3, State: Initialized, Role: LEARNER
I20240503 08:12:25.634506 27699 consensus_queue.cc:260] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 349, Last appended: 1.349, Last appended by leader: 349, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } }
I20240503 08:12:25.636029 27699 ts_tablet_manager.cc:1398] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:25.637369 24537 tablet_copy_service.cc:342] P 6a640901590545eeb69e216d2399ea51: Request end of tablet copy session efe9f2f995024b01b2ade8bdc404a2a3-3071edf0c1004a6496d1cb46591d422e received from {username='slave'} at 127.20.107.136:56121
I20240503 08:12:25.637526 24537 tablet_copy_service.cc:433] P 6a640901590545eeb69e216d2399ea51: ending tablet copy session efe9f2f995024b01b2ade8bdc404a2a3-3071edf0c1004a6496d1cb46591d422e on tablet 3071edf0c1004a6496d1cb46591d422e with peer efe9f2f995024b01b2ade8bdc404a2a3
I20240503 08:12:25.678877 27732 tablet_bootstrap.cc:492] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=349 overwritten=0 applied=349 ignored=0} inserts{seen=1021 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:25.679294 27732 tablet_bootstrap.cc:492] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3: Bootstrap complete.
I20240503 08:12:25.679574 27732 ts_tablet_manager.cc:1367] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3: Time spent bootstrapping tablet: real 0.468s user 0.386s sys 0.038s
I20240503 08:12:25.680227 27732 raft_consensus.cc:348] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } }
I20240503 08:12:25.680436 27732 raft_consensus.cc:729] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efe9f2f995024b01b2ade8bdc404a2a3, State: Initialized, Role: LEARNER
I20240503 08:12:25.680634 27732 consensus_queue.cc:260] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 349, Last appended: 1.349, Last appended by leader: 349, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } }
I20240503 08:12:25.681202 27732 ts_tablet_manager.cc:1398] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:12:25.682008 24406 tablet_copy_service.cc:342] P b53502e2a1ae4c86a054d38048dd0df6: Request end of tablet copy session efe9f2f995024b01b2ade8bdc404a2a3-52edf3b518be4f45bb347dbbe4bf7fd9 received from {username='slave'} at 127.20.107.136:43581
I20240503 08:12:25.682186 24406 tablet_copy_service.cc:433] P b53502e2a1ae4c86a054d38048dd0df6: ending tablet copy session efe9f2f995024b01b2ade8bdc404a2a3-52edf3b518be4f45bb347dbbe4bf7fd9 on tablet 52edf3b518be4f45bb347dbbe4bf7fd9 with peer efe9f2f995024b01b2ade8bdc404a2a3
I20240503 08:12:25.747175 27731 tablet_bootstrap.cc:492] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=378 overwritten=0 applied=378 ignored=0} inserts{seen=1006 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:25.747566 27731 tablet_bootstrap.cc:492] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3: Bootstrap complete.
I20240503 08:12:25.747845 27731 ts_tablet_manager.cc:1367] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3: Time spent bootstrapping tablet: real 0.537s user 0.412s sys 0.034s
I20240503 08:12:25.748440 27731 raft_consensus.cc:348] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } }
I20240503 08:12:25.748625 27731 raft_consensus.cc:729] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efe9f2f995024b01b2ade8bdc404a2a3, State: Initialized, Role: LEARNER
I20240503 08:12:25.748767 27731 consensus_queue.cc:260] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 378, Last appended: 1.378, Last appended by leader: 378, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: true } }
I20240503 08:12:25.749301 27731 ts_tablet_manager.cc:1398] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:12:25.750064 24667 tablet_copy_service.cc:342] P 56fb8e40a4da4b3794f07ad8137ecf42: Request end of tablet copy session efe9f2f995024b01b2ade8bdc404a2a3-d73b0d5398cd484ba45515cf95cd5b14 received from {username='slave'} at 127.20.107.136:43169
I20240503 08:12:25.750205 24667 tablet_copy_service.cc:433] P 56fb8e40a4da4b3794f07ad8137ecf42: ending tablet copy session efe9f2f995024b01b2ade8bdc404a2a3-d73b0d5398cd484ba45515cf95cd5b14 on tablet d73b0d5398cd484ba45515cf95cd5b14 with peer efe9f2f995024b01b2ade8bdc404a2a3
I20240503 08:12:25.972816 27481 raft_consensus.cc:1203] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.348->[1.349-1.349] Dedup: 1.349->[]
I20240503 08:12:26.051111 27481 raft_consensus.cc:1203] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.348->[1.349-1.349] Dedup: 1.349->[]
I20240503 08:12:26.171532 27816 raft_consensus.cc:1050] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e: attempting to promote NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 to VOTER
I20240503 08:12:26.172191 27816 consensus_queue.cc:237] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:26.174613 25961 raft_consensus.cc:1261] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Refusing update from remote peer a0d221e07029452cae34b8920884507e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:12:26.174654 27481 raft_consensus.cc:1261] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 LEARNER]: Refusing update from remote peer a0d221e07029452cae34b8920884507e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:12:26.175271 27818 consensus_queue.cc:1035] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:26.175669 27816 consensus_queue.cc:1035] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:26.179376 25961 raft_consensus.cc:2932] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.178738 27818 raft_consensus.cc:2932] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.180318 27481 raft_consensus.cc:2932] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.184902 24236 catalog_manager.cc:5484] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e reported cstate change: config changed from index 6 to 7, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a0d221e07029452cae34b8920884507e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:26.190172 26090 consensus_queue.cc:237] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:26.191638 27481 raft_consensus.cc:1261] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 FOLLOWER]: Refusing update from remote peer a0d221e07029452cae34b8920884507e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:12:26.192118 27818 consensus_queue.cc:1035] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:12:26.194130 27816 raft_consensus.cc:2932] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.194684 27481 raft_consensus.cc:2932] T 6d422797de9843edbc473fca1d8e9e87 P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.198036 24231 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6d422797de9843edbc473fca1d8e9e87 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:26.198355 24242 catalog_manager.cc:5484] T 6d422797de9843edbc473fca1d8e9e87 P a0d221e07029452cae34b8920884507e reported cstate change: config changed from index 7 to 8, VOTER 9d313001b0a647caba8f53540a2b67ac (127.20.107.134) evicted. New cstate: current_term: 2 leader_uuid: "a0d221e07029452cae34b8920884507e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:26.215735 25941 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d422797de9843edbc473fca1d8e9e87 with delete_type TABLET_DATA_TOMBSTONED (TS 9d313001b0a647caba8f53540a2b67ac not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:54638
I20240503 08:12:26.216678 27828 tablet_replica.cc:330] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac: stopping tablet replica
I20240503 08:12:26.217133 27828 raft_consensus.cc:2229] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:26.217540 27828 raft_consensus.cc:2258] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:26.219708 27828 ts_tablet_manager.cc:1875] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:26.224365 27828 ts_tablet_manager.cc:1888] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 08:12:26.224484 27828 log.cc:1196] T 6d422797de9843edbc473fca1d8e9e87 P 9d313001b0a647caba8f53540a2b67ac: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal/wals/6d422797de9843edbc473fca1d8e9e87
I20240503 08:12:26.225371 24230 catalog_manager.cc:4830] TS 9d313001b0a647caba8f53540a2b67ac (127.20.107.134:46393): tablet 6d422797de9843edbc473fca1d8e9e87 (table TestTable [id=072979b49b8b4071923d58835e04c91c]) successfully deleted
I20240503 08:12:26.243257 27481 raft_consensus.cc:1203] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.377->[1.378-1.378] Dedup: 1.378->[]
I20240503 08:12:26.432960 27804 raft_consensus.cc:1050] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51: attempting to promote NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 to VOTER
I20240503 08:12:26.433630 27804 consensus_queue.cc:237] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 1.349, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:26.435528 27481 raft_consensus.cc:1261] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 LEARNER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240503 08:12:26.435607 25697 raft_consensus.cc:1261] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240503 08:12:26.436203 27805 consensus_queue.cc:1035] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20240503 08:12:26.436578 27762 consensus_queue.cc:1035] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20240503 08:12:26.439201 27805 raft_consensus.cc:2932] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Committing config change with OpId 1.350: config changed from index 349 to 350, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.439872 25697 raft_consensus.cc:2932] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 349 to 350, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.440287 27481 raft_consensus.cc:2932] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 349 to 350, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.444224 24240 catalog_manager.cc:5484] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 reported cstate change: config changed from index 349 to 350, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.496112 27870 raft_consensus.cc:1050] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6: attempting to promote NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 to VOTER
I20240503 08:12:26.496707 27870 consensus_queue.cc:237] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 1.349, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:26.498574 27481 raft_consensus.cc:1261] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 LEARNER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240503 08:12:26.498739 24515 raft_consensus.cc:1261] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Refusing update from remote peer b53502e2a1ae4c86a054d38048dd0df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240503 08:12:26.499078 27872 consensus_queue.cc:1035] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20240503 08:12:26.499406 27871 consensus_queue.cc:1035] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20240503 08:12:26.501855 27872 raft_consensus.cc:2932] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 LEADER]: Committing config change with OpId 1.350: config changed from index 349 to 350, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.503655 24515 raft_consensus.cc:2932] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 349 to 350, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.505297 27481 raft_consensus.cc:2932] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 349 to 350, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.508085 24236 catalog_manager.cc:5484] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 reported cstate change: config changed from index 349 to 350, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b53502e2a1ae4c86a054d38048dd0df6" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.575748 24383 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "52edf3b518be4f45bb347dbbe4bf7fd9"
dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46566
I20240503 08:12:26.576002 24383 raft_consensus.cc:595] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:26.594615 24515 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3071edf0c1004a6496d1cb46591d422e"
dest_uuid: "6a640901590545eeb69e216d2399ea51"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38502
I20240503 08:12:26.594869 24515 raft_consensus.cc:595] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:26.824339 24383 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "52edf3b518be4f45bb347dbbe4bf7fd9"
dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46584
I20240503 08:12:26.824558 24383 raft_consensus.cc:595] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:26.843704 24515 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3071edf0c1004a6496d1cb46591d422e"
dest_uuid: "6a640901590545eeb69e216d2399ea51"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38530
I20240503 08:12:26.843968 24515 raft_consensus.cc:595] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:26.855723 27927 raft_consensus.cc:1050] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42: attempting to promote NON_VOTER efe9f2f995024b01b2ade8bdc404a2a3 to VOTER
I20240503 08:12:26.856639 27927 consensus_queue.cc:237] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 378, Committed index: 378, Last appended: 1.378, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:26.858803 26090 raft_consensus.cc:1261] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 378. Preceding OpId from leader: term: 1 index: 379. (index mismatch)
I20240503 08:12:26.858810 27481 raft_consensus.cc:1261] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 LEARNER]: Refusing update from remote peer 56fb8e40a4da4b3794f07ad8137ecf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 378. Preceding OpId from leader: term: 1 index: 379. (index mismatch)
I20240503 08:12:26.859364 27928 consensus_queue.cc:1035] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 379, Last known committed idx: 378, Time since last communication: 0.000s
I20240503 08:12:26.859792 27830 consensus_queue.cc:1035] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 379, Last known committed idx: 378, Time since last communication: 0.000s
I20240503 08:12:26.862493 27928 raft_consensus.cc:2932] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Committing config change with OpId 1.379: config changed from index 378 to 379, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.863201 26090 raft_consensus.cc:2932] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Committing config change with OpId 1.379: config changed from index 378 to 379, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.863314 27481 raft_consensus.cc:2932] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 FOLLOWER]: Committing config change with OpId 1.379: config changed from index 378 to 379, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:26.868295 24242 catalog_manager.cc:5484] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 reported cstate change: config changed from index 378 to 379, efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" committed_config { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:26.935497 27805 raft_consensus.cc:979] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51: : Instructing follower e7416e7c03a74a24b8fe2508111770d7 to start an election
I20240503 08:12:26.935689 27805 raft_consensus.cc:1067] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Signalling peer e7416e7c03a74a24b8fe2508111770d7 to start an election
I20240503 08:12:26.936308 25697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3071edf0c1004a6496d1cb46591d422e"
dest_uuid: "e7416e7c03a74a24b8fe2508111770d7"
from {username='slave'} at 127.20.107.130:59965
I20240503 08:12:26.936519 25697 raft_consensus.cc:482] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:26.936635 25697 raft_consensus.cc:3037] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:26.938570 25697 raft_consensus.cc:504] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:26.939383 25697 leader_election.cc:288] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [CANDIDATE]: Term 2 election: Requested vote from peers 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793), efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136:41565)
I20240503 08:12:26.939787 24515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3071edf0c1004a6496d1cb46591d422e" candidate_uuid: "e7416e7c03a74a24b8fe2508111770d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 350 } } ignore_live_leader: true dest_uuid: "6a640901590545eeb69e216d2399ea51"
I20240503 08:12:26.940048 24515 raft_consensus.cc:3032] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:26.940145 24515 raft_consensus.cc:729] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Running, Role: LEADER
I20240503 08:12:26.940461 24515 consensus_queue.cc:260] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 1.350, Last appended by leader: 350, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:26.940837 24515 raft_consensus.cc:3037] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:26.943504 24515 raft_consensus.cc:2454] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7416e7c03a74a24b8fe2508111770d7 in term 2.
I20240503 08:12:26.943985 25633 leader_election.cc:302] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, e7416e7c03a74a24b8fe2508111770d7; no voters:
I20240503 08:12:26.944331 27831 raft_consensus.cc:2781] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:26.944759 27831 raft_consensus.cc:686] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 2 LEADER]: Becoming Leader. State: Replica: e7416e7c03a74a24b8fe2508111770d7, State: Running, Role: LEADER
I20240503 08:12:26.945161 27831 consensus_queue.cc:237] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 1.350, Last appended by leader: 350, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:26.946533 27481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3071edf0c1004a6496d1cb46591d422e" candidate_uuid: "e7416e7c03a74a24b8fe2508111770d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 350 } } ignore_live_leader: true dest_uuid: "efe9f2f995024b01b2ade8bdc404a2a3"
I20240503 08:12:26.946872 27481 raft_consensus.cc:3037] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:26.948148 24236 catalog_manager.cc:5484] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 reported cstate change: term changed from 1 to 2, leader changed from 6a640901590545eeb69e216d2399ea51 (127.20.107.130) to e7416e7c03a74a24b8fe2508111770d7 (127.20.107.132). New cstate: current_term: 2 leader_uuid: "e7416e7c03a74a24b8fe2508111770d7" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:26.949699 27481 raft_consensus.cc:2454] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7416e7c03a74a24b8fe2508111770d7 in term 2.
I20240503 08:12:27.024960 27935 raft_consensus.cc:979] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6: : Instructing follower 6a640901590545eeb69e216d2399ea51 to start an election
I20240503 08:12:27.025169 27935 raft_consensus.cc:1067] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 LEADER]: Signalling peer 6a640901590545eeb69e216d2399ea51 to start an election
I20240503 08:12:27.025836 24515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "52edf3b518be4f45bb347dbbe4bf7fd9"
dest_uuid: "6a640901590545eeb69e216d2399ea51"
from {username='slave'} at 127.20.107.129:44795
I20240503 08:12:27.026069 24515 raft_consensus.cc:482] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:27.026180 24515 raft_consensus.cc:3037] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:27.028352 24515 raft_consensus.cc:504] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:27.029182 24515 leader_election.cc:288] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 election: Requested vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775), efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136:41565)
I20240503 08:12:27.029621 24383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52edf3b518be4f45bb347dbbe4bf7fd9" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 2 candidate_status { last_received { term: 1 index: 350 } } ignore_live_leader: true dest_uuid: "b53502e2a1ae4c86a054d38048dd0df6"
I20240503 08:12:27.029870 24383 raft_consensus.cc:3032] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:27.029820 27481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52edf3b518be4f45bb347dbbe4bf7fd9" candidate_uuid: "6a640901590545eeb69e216d2399ea51" candidate_term: 2 candidate_status { last_received { term: 1 index: 350 } } ignore_live_leader: true dest_uuid: "efe9f2f995024b01b2ade8bdc404a2a3"
I20240503 08:12:27.029959 24383 raft_consensus.cc:729] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b53502e2a1ae4c86a054d38048dd0df6, State: Running, Role: LEADER
I20240503 08:12:27.030040 27481 raft_consensus.cc:3037] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:27.030277 24383 consensus_queue.cc:260] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 1.350, Last appended by leader: 350, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:27.030656 24383 raft_consensus.cc:3037] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:27.032824 27481 raft_consensus.cc:2454] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a640901590545eeb69e216d2399ea51 in term 2.
I20240503 08:12:27.033432 24383 raft_consensus.cc:2454] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a640901590545eeb69e216d2399ea51 in term 2.
I20240503 08:12:27.033454 24450 leader_election.cc:302] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a640901590545eeb69e216d2399ea51, efe9f2f995024b01b2ade8bdc404a2a3; no voters:
I20240503 08:12:27.033795 27805 raft_consensus.cc:2781] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:27.033957 27805 raft_consensus.cc:686] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 2 LEADER]: Becoming Leader. State: Replica: 6a640901590545eeb69e216d2399ea51, State: Running, Role: LEADER
I20240503 08:12:27.034250 27805 consensus_queue.cc:237] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 1.350, Last appended by leader: 350, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:27.036713 24236 catalog_manager.cc:5484] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 reported cstate change: term changed from 1 to 2, leader changed from b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) to 6a640901590545eeb69e216d2399ea51 (127.20.107.130). New cstate: current_term: 2 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:27.063179 24646 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d73b0d5398cd484ba45515cf95cd5b14"
dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53932
I20240503 08:12:27.063433 24646 raft_consensus.cc:595] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:27.301962 24646 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d73b0d5398cd484ba45515cf95cd5b14"
dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53952
I20240503 08:12:27.302186 24646 raft_consensus.cc:595] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:27.336786 27928 raft_consensus.cc:979] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42: : Instructing follower a0d221e07029452cae34b8920884507e to start an election
I20240503 08:12:27.336995 27830 raft_consensus.cc:1067] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Signalling peer a0d221e07029452cae34b8920884507e to start an election
I20240503 08:12:27.337632 26090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d73b0d5398cd484ba45515cf95cd5b14"
dest_uuid: "a0d221e07029452cae34b8920884507e"
from {username='slave'} at 127.20.107.131:35013
I20240503 08:12:27.337854 26090 raft_consensus.cc:482] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:27.337945 26090 raft_consensus.cc:3037] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:27.340269 26090 raft_consensus.cc:504] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:27.341024 26090 leader_election.cc:288] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [CANDIDATE]: Term 2 election: Requested vote from peers 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003), efe9f2f995024b01b2ade8bdc404a2a3 (127.20.107.136:41565)
I20240503 08:12:27.341363 24646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73b0d5398cd484ba45515cf95cd5b14" candidate_uuid: "a0d221e07029452cae34b8920884507e" candidate_term: 2 candidate_status { last_received { term: 1 index: 379 } } ignore_live_leader: true dest_uuid: "56fb8e40a4da4b3794f07ad8137ecf42"
I20240503 08:12:27.341601 24646 raft_consensus.cc:3032] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:27.341569 27481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73b0d5398cd484ba45515cf95cd5b14" candidate_uuid: "a0d221e07029452cae34b8920884507e" candidate_term: 2 candidate_status { last_received { term: 1 index: 379 } } ignore_live_leader: true dest_uuid: "efe9f2f995024b01b2ade8bdc404a2a3"
I20240503 08:12:27.341687 24646 raft_consensus.cc:729] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 56fb8e40a4da4b3794f07ad8137ecf42, State: Running, Role: LEADER
I20240503 08:12:27.341751 27481 raft_consensus.cc:3037] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:27.341967 24646 consensus_queue.cc:260] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 379, Committed index: 379, Last appended: 1.379, Last appended by leader: 379, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:27.342296 24646 raft_consensus.cc:3037] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:27.343653 27481 raft_consensus.cc:2454] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d221e07029452cae34b8920884507e in term 2.
I20240503 08:12:27.344053 26024 leader_election.cc:302] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d221e07029452cae34b8920884507e, efe9f2f995024b01b2ade8bdc404a2a3; no voters:
I20240503 08:12:27.344316 27818 raft_consensus.cc:2781] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:27.344719 27818 raft_consensus.cc:686] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 2 LEADER]: Becoming Leader. State: Replica: a0d221e07029452cae34b8920884507e, State: Running, Role: LEADER
I20240503 08:12:27.344728 24646 raft_consensus.cc:2454] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d221e07029452cae34b8920884507e in term 2.
I20240503 08:12:27.345016 27818 consensus_queue.cc:237] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 379, Committed index: 379, Last appended: 1.379, Last appended by leader: 379, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:27.347397 24236 catalog_manager.cc:5484] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e reported cstate change: term changed from 1 to 2, leader changed from 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) to a0d221e07029452cae34b8920884507e (127.20.107.135). New cstate: current_term: 2 leader_uuid: "a0d221e07029452cae34b8920884507e" committed_config { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:27.429365 24515 raft_consensus.cc:1261] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Refusing update from remote peer e7416e7c03a74a24b8fe2508111770d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 2 index: 351. (index mismatch)
I20240503 08:12:27.429957 27831 consensus_queue.cc:1035] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240503 08:12:27.434648 27481 raft_consensus.cc:1261] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 FOLLOWER]: Refusing update from remote peer e7416e7c03a74a24b8fe2508111770d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 2 index: 351. (index mismatch)
I20240503 08:12:27.435243 27981 consensus_queue.cc:1035] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240503 08:12:27.463904 24383 raft_consensus.cc:1261] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 2 index: 351. (index mismatch)
I20240503 08:12:27.464543 27805 consensus_queue.cc:1035] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240503 08:12:27.469451 27481 raft_consensus.cc:1261] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 2 index: 351. (index mismatch)
I20240503 08:12:27.470472 27762 consensus_queue.cc:1035] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240503 08:12:27.830132 24646 raft_consensus.cc:1261] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Refusing update from remote peer a0d221e07029452cae34b8920884507e: Log matching property violated. Preceding OpId in replica: term: 1 index: 379. Preceding OpId from leader: term: 2 index: 380. (index mismatch)
I20240503 08:12:27.830750 27818 consensus_queue.cc:1035] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fb8e40a4da4b3794f07ad8137ecf42" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 34003 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 380, Last known committed idx: 379, Time since last communication: 0.000s
I20240503 08:12:27.835186 27481 raft_consensus.cc:1261] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 FOLLOWER]: Refusing update from remote peer a0d221e07029452cae34b8920884507e: Log matching property violated. Preceding OpId in replica: term: 1 index: 379. Preceding OpId from leader: term: 2 index: 380. (index mismatch)
I20240503 08:12:27.835860 28019 consensus_queue.cc:1035] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 380, Last known committed idx: 379, Time since last communication: 0.000s
I20240503 08:12:27.870769 24515 consensus_queue.cc:237] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 2.351, Last appended by leader: 350, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:27.872390 27481 raft_consensus.cc:1261] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 FOLLOWER]: Refusing update from remote peer 6a640901590545eeb69e216d2399ea51: Log matching property violated. Preceding OpId in replica: term: 2 index: 351. Preceding OpId from leader: term: 2 index: 352. (index mismatch)
I20240503 08:12:27.873003 27984 consensus_queue.cc:1035] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20240503 08:12:27.875430 27805 raft_consensus.cc:2932] T 52edf3b518be4f45bb347dbbe4bf7fd9 P 6a640901590545eeb69e216d2399ea51 [term 2 LEADER]: Committing config change with OpId 2.352: config changed from index 350 to 352, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:27.875988 27481 raft_consensus.cc:2932] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 FOLLOWER]: Committing config change with OpId 2.352: config changed from index 350 to 352, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:27.879297 24230 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 52edf3b518be4f45bb347dbbe4bf7fd9 with cas_config_opid_index 350: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:27.881367 24236 catalog_manager.cc:5484] T 52edf3b518be4f45bb347dbbe4bf7fd9 P efe9f2f995024b01b2ade8bdc404a2a3 reported cstate change: config changed from index 350 to 352, VOTER b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129) evicted. New cstate: current_term: 2 leader_uuid: "6a640901590545eeb69e216d2399ea51" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "6a640901590545eeb69e216d2399ea51" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 43793 } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:27.885761 24364 tablet_service.cc:1514] Processing DeleteTablet for tablet 52edf3b518be4f45bb347dbbe4bf7fd9 with delete_type TABLET_DATA_TOMBSTONED (TS b53502e2a1ae4c86a054d38048dd0df6 not found in new config with opid_index 352) from {username='slave'} at 127.0.0.1:38614
I20240503 08:12:27.886399 28023 tablet_replica.cc:330] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6: stopping tablet replica
I20240503 08:12:27.886622 28023 raft_consensus.cc:2229] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:27.886838 28023 raft_consensus.cc:2258] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:27.888346 28023 ts_tablet_manager.cc:1875] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:27.892782 28023 ts_tablet_manager.cc:1888] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.351
I20240503 08:12:27.892953 28023 log.cc:1196] T 52edf3b518be4f45bb347dbbe4bf7fd9 P b53502e2a1ae4c86a054d38048dd0df6: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/52edf3b518be4f45bb347dbbe4bf7fd9
I20240503 08:12:27.893725 24229 catalog_manager.cc:4830] TS b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775): tablet 52edf3b518be4f45bb347dbbe4bf7fd9 (table rebalance_test_table_2 [id=4f8b8cf1869a436f9c012044eb10dc54]) successfully deleted
I20240503 08:12:27.958884 25697 consensus_queue.cc:237] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 2.351, Last appended by leader: 350, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:27.960698 27481 raft_consensus.cc:1261] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 FOLLOWER]: Refusing update from remote peer e7416e7c03a74a24b8fe2508111770d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 351. Preceding OpId from leader: term: 2 index: 352. (index mismatch)
I20240503 08:12:27.961341 28025 consensus_queue.cc:1035] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20240503 08:12:27.963812 28024 raft_consensus.cc:2932] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 [term 2 LEADER]: Committing config change with OpId 2.352: config changed from index 350 to 352, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:27.964479 27481 raft_consensus.cc:2932] T 3071edf0c1004a6496d1cb46591d422e P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 FOLLOWER]: Committing config change with OpId 2.352: config changed from index 350 to 352, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:27.968221 24230 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3071edf0c1004a6496d1cb46591d422e with cas_config_opid_index 350: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:27.969743 24240 catalog_manager.cc:5484] T 3071edf0c1004a6496d1cb46591d422e P e7416e7c03a74a24b8fe2508111770d7 reported cstate change: config changed from index 350 to 352, VOTER 6a640901590545eeb69e216d2399ea51 (127.20.107.130) evicted. New cstate: current_term: 2 leader_uuid: "e7416e7c03a74a24b8fe2508111770d7" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "e7416e7c03a74a24b8fe2508111770d7" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 36475 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:27.973562 24496 tablet_service.cc:1514] Processing DeleteTablet for tablet 3071edf0c1004a6496d1cb46591d422e with delete_type TABLET_DATA_TOMBSTONED (TS 6a640901590545eeb69e216d2399ea51 not found in new config with opid_index 352) from {username='slave'} at 127.0.0.1:42320
I20240503 08:12:27.974336 28026 tablet_replica.cc:330] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51: stopping tablet replica
I20240503 08:12:27.974536 28026 raft_consensus.cc:2229] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:27.974833 28026 raft_consensus.cc:2258] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:27.977249 28026 ts_tablet_manager.cc:1875] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:27.982671 28026 ts_tablet_manager.cc:1888] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.351
I20240503 08:12:27.982808 28026 log.cc:1196] T 3071edf0c1004a6496d1cb46591d422e P 6a640901590545eeb69e216d2399ea51: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/3071edf0c1004a6496d1cb46591d422e
I20240503 08:12:27.983641 24230 catalog_manager.cc:4830] TS 6a640901590545eeb69e216d2399ea51 (127.20.107.130:43793): tablet 3071edf0c1004a6496d1cb46591d422e (table rebalance_test_table_3 [id=e6ccc802e346493aa6739ba0f901f14d]) successfully deleted
I20240503 08:12:28.357054 26090 consensus_queue.cc:237] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 380, Committed index: 380, Last appended: 2.380, Last appended by leader: 379, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } }
I20240503 08:12:28.358512 27481 raft_consensus.cc:1261] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 FOLLOWER]: Refusing update from remote peer a0d221e07029452cae34b8920884507e: Log matching property violated. Preceding OpId in replica: term: 2 index: 380. Preceding OpId from leader: term: 2 index: 381. (index mismatch)
I20240503 08:12:28.359069 28052 consensus_queue.cc:1035] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 381, Last known committed idx: 380, Time since last communication: 0.000s
I20240503 08:12:28.361409 28053 raft_consensus.cc:2932] T d73b0d5398cd484ba45515cf95cd5b14 P a0d221e07029452cae34b8920884507e [term 2 LEADER]: Committing config change with OpId 2.381: config changed from index 379 to 381, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New config: { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:28.362354 27481 raft_consensus.cc:2932] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3 [term 2 FOLLOWER]: Committing config change with OpId 2.381: config changed from index 379 to 381, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New config: { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:28.365051 24231 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d73b0d5398cd484ba45515cf95cd5b14 with cas_config_opid_index 379: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:28.366966 24236 catalog_manager.cc:5484] T d73b0d5398cd484ba45515cf95cd5b14 P efe9f2f995024b01b2ade8bdc404a2a3 reported cstate change: config changed from index 379 to 381, VOTER 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131) evicted. New cstate: current_term: 2 leader_uuid: "a0d221e07029452cae34b8920884507e" committed_config { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "a0d221e07029452cae34b8920884507e" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45329 } attrs { promote: false } } peers { permanent_uuid: "efe9f2f995024b01b2ade8bdc404a2a3" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 41565 } attrs { promote: false } } }
I20240503 08:12:28.371055 24627 tablet_service.cc:1514] Processing DeleteTablet for tablet d73b0d5398cd484ba45515cf95cd5b14 with delete_type TABLET_DATA_TOMBSTONED (TS 56fb8e40a4da4b3794f07ad8137ecf42 not found in new config with opid_index 381) from {username='slave'} at 127.0.0.1:46200
I20240503 08:12:28.371721 28054 tablet_replica.cc:330] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42: stopping tablet replica
I20240503 08:12:28.371886 28054 raft_consensus.cc:2229] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:28.372042 28054 raft_consensus.cc:2258] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:28.373643 28054 ts_tablet_manager.cc:1875] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:28.378222 28054 ts_tablet_manager.cc:1888] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.380
I20240503 08:12:28.378391 28054 log.cc:1196] T d73b0d5398cd484ba45515cf95cd5b14 P 56fb8e40a4da4b3794f07ad8137ecf42: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/d73b0d5398cd484ba45515cf95cd5b14
I20240503 08:12:28.379163 24230 catalog_manager.cc:4830] TS 56fb8e40a4da4b3794f07ad8137ecf42 (127.20.107.131:34003): tablet d73b0d5398cd484ba45515cf95cd5b14 (table rebalance_test_table_4 [id=72d93aae96774cf887b14b98c748dffc]) successfully deleted
I20240503 08:12:28.622066 24496 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:28.626513 24627 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:28.636058 24364 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:28.637174 26072 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:28.648000 25941 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:28.664892 25679 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:28.681641 27461 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:28.703845 25811 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:28.895678 24627 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:28.900231 24496 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:28.900727 25941 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:28.902163 26072 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:28.924094 24364 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:28.934708 25679 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:28.941233 25811 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:28.942265 27461 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:29.390467 24627 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:29.399554 26072 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:29.401449 25941 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:29.402066 24496 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:29.404937 24364 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:29.428336 25679 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:29.429244 27461 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:29.525102 25811 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
4a981765e30a453eab8294095153bf23 | 127.20.107.190:44243 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.107.148:36581 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
56fb8e40a4da4b3794f07ad8137ecf42 | 127.20.107.131:34003 | HEALTHY | <none> | 3 | 0
6a640901590545eeb69e216d2399ea51 | 127.20.107.130:43793 | HEALTHY | <none> | 3 | 0
9d313001b0a647caba8f53540a2b67ac | 127.20.107.134:46393 | HEALTHY | <none> | 0 | 0
a0d221e07029452cae34b8920884507e | 127.20.107.135:45329 | HEALTHY | <none> | 2 | 0
b53502e2a1ae4c86a054d38048dd0df6 | 127.20.107.129:46775 | HEALTHY | <none> | 4 | 0
e7416e7c03a74a24b8fe2508111770d7 | 127.20.107.132:36475 | HEALTHY | <none> | 2 | 0
efe9f2f995024b01b2ade8bdc404a2a3 | 127.20.107.136:41565 | HEALTHY | <none> | 0 | 0
f9705fdde3854639ba85097a161f37e7 | 127.20.107.133:45569 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.20.107.129 | experimental | 127.20.107.129:46775
local_ip_for_outbound_sockets | 127.20.107.130 | experimental | 127.20.107.130:43793
local_ip_for_outbound_sockets | 127.20.107.131 | experimental | 127.20.107.131:34003
local_ip_for_outbound_sockets | 127.20.107.132 | experimental | 127.20.107.132:36475
local_ip_for_outbound_sockets | 127.20.107.133 | experimental | 127.20.107.133:45569
local_ip_for_outbound_sockets | 127.20.107.134 | experimental | 127.20.107.134:46393
local_ip_for_outbound_sockets | 127.20.107.135 | experimental | 127.20.107.135:45329
local_ip_for_outbound_sockets | 127.20.107.136 | experimental | 127.20.107.136:41565
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.20.107.129:46775
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.20.107.130:43793
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.20.107.131:34003
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.20.107.132:36475
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.20.107.133:45569
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.20.107.134:46393
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.20.107.135:45329
server_dump_info_path | /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.20.107.136:41565
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.107.148:36581 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:12:29.570063 20910 log_verifier.cc:126] Checking tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1
I20240503 08:12:29.714601 20910 log_verifier.cc:177] Verified matching terms for 349 ops in tablet 08df2ee2ca344cd2abd5ecb67ba9ffc1
I20240503 08:12:29.715129 20910 log_verifier.cc:126] Checking tablet 3071edf0c1004a6496d1cb46591d422e
I20240503 08:12:29.839390 20910 log_verifier.cc:177] Verified matching terms for 352 ops in tablet 3071edf0c1004a6496d1cb46591d422e
I20240503 08:12:29.839928 20910 log_verifier.cc:126] Checking tablet 39ccbb3b60094ed1a8a09ae6b6c4e6a5
I20240503 08:12:29.974720 20910 log_verifier.cc:177] Verified matching terms for 379 ops in tablet 39ccbb3b60094ed1a8a09ae6b6c4e6a5
I20240503 08:12:29.975240 20910 log_verifier.cc:126] Checking tablet 3c65da71faf34351885b5033d3d68835
I20240503 08:12:30.118943 20910 log_verifier.cc:177] Verified matching terms for 347 ops in tablet 3c65da71faf34351885b5033d3d68835
I20240503 08:12:30.119426 20910 log_verifier.cc:126] Checking tablet 52edf3b518be4f45bb347dbbe4bf7fd9
I20240503 08:12:30.262243 20910 log_verifier.cc:177] Verified matching terms for 352 ops in tablet 52edf3b518be4f45bb347dbbe4bf7fd9
I20240503 08:12:30.262768 20910 log_verifier.cc:126] Checking tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0
I20240503 08:12:30.391530 20910 log_verifier.cc:177] Verified matching terms for 374 ops in tablet 5dd5613d173f4b1eb0eacf2fec3a3ff0
I20240503 08:12:30.392035 20910 log_verifier.cc:126] Checking tablet 61bf5a9f26034821b821a45ac37ba133
I20240503 08:12:30.531105 20910 log_verifier.cc:177] Verified matching terms for 374 ops in tablet 61bf5a9f26034821b821a45ac37ba133
I20240503 08:12:30.531620 20910 log_verifier.cc:126] Checking tablet 6d422797de9843edbc473fca1d8e9e87
I20240503 08:12:30.546475 20910 log_verifier.cc:177] Verified matching terms for 8 ops in tablet 6d422797de9843edbc473fca1d8e9e87
I20240503 08:12:30.546602 20910 log_verifier.cc:126] Checking tablet 7b858bcd6e1c469d9f19f142106908d3
I20240503 08:12:30.668900 20910 log_verifier.cc:177] Verified matching terms for 341 ops in tablet 7b858bcd6e1c469d9f19f142106908d3
I20240503 08:12:30.669394 20910 log_verifier.cc:126] Checking tablet 7d2772d08c06498db2cb0b02497ea042
I20240503 08:12:30.798601 20910 log_verifier.cc:177] Verified matching terms for 330 ops in tablet 7d2772d08c06498db2cb0b02497ea042
I20240503 08:12:30.799206 20910 log_verifier.cc:126] Checking tablet 83697ce9030f4981af717d18eb6757a4
I20240503 08:12:30.935966 20910 log_verifier.cc:177] Verified matching terms for 379 ops in tablet 83697ce9030f4981af717d18eb6757a4
I20240503 08:12:30.936451 20910 log_verifier.cc:126] Checking tablet c85cf905ebfa4e7b91a0af9ed2fa77da
I20240503 08:12:31.072016 20910 log_verifier.cc:177] Verified matching terms for 370 ops in tablet c85cf905ebfa4e7b91a0af9ed2fa77da
I20240503 08:12:31.072499 20910 log_verifier.cc:126] Checking tablet ca9558ccb5444f61b3ee86625b2f1097
I20240503 08:12:31.188509 20910 log_verifier.cc:177] Verified matching terms for 316 ops in tablet ca9558ccb5444f61b3ee86625b2f1097
I20240503 08:12:31.188966 20910 log_verifier.cc:126] Checking tablet d73b0d5398cd484ba45515cf95cd5b14
I20240503 08:12:31.330581 20910 log_verifier.cc:177] Verified matching terms for 381 ops in tablet d73b0d5398cd484ba45515cf95cd5b14
I20240503 08:12:31.331092 20910 log_verifier.cc:126] Checking tablet d8a32af7adaf410582e1b898257db62d
I20240503 08:12:31.483990 20910 log_verifier.cc:177] Verified matching terms for 397 ops in tablet d8a32af7adaf410582e1b898257db62d
I20240503 08:12:31.484503 20910 log_verifier.cc:126] Checking tablet e0756a24eb5d4e26b2d44e46638a74be
I20240503 08:12:31.612296 20910 log_verifier.cc:177] Verified matching terms for 352 ops in tablet e0756a24eb5d4e26b2d44e46638a74be
I20240503 08:12:31.619371 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 24304
I20240503 08:12:31.620893 24426 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:31.858932 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 24304
I20240503 08:12:31.902310 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 24435
W20240503 08:12:31.902619 24450 connection.cc:535] server connection from 127.20.107.129:44795 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:12:31.903419 24556 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:32.133445 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 24435
W20240503 08:12:32.189256 26027 connection.cc:535] server connection from 127.20.107.130:60433 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:12:32.189643 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 24565
W20240503 08:12:32.189707 27415 connection.cc:535] server connection from 127.20.107.130:52961 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:12:32.189785 25896 connection.cc:535] server connection from 127.20.107.130:60767 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:12:32.190671 24686 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:32.418463 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 24565
I20240503 08:12:32.460165 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 25615
I20240503 08:12:32.466625 25738 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:32.629865 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 25615
W20240503 08:12:32.659773 27415 connection.cc:535] server connection from 127.20.107.132:40305 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:12:32.660194 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 25748
I20240503 08:12:32.666620 25871 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:32.829782 27972 raft_consensus.cc:482] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b53502e2a1ae4c86a054d38048dd0df6)
I20240503 08:12:32.829991 27972 raft_consensus.cc:504] T 83697ce9030f4981af717d18eb6757a4 P f9705fdde3854639ba85097a161f37e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "f9705fdde3854639ba85097a161f37e7" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 45569 } attrs { promote: false } }
I20240503 08:12:32.830307 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 25748
I20240503 08:12:32.863313 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 25879
I20240503 08:12:32.869647 26000 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:33.032930 28187 raft_consensus.cc:482] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b53502e2a1ae4c86a054d38048dd0df6)
I20240503 08:12:33.033185 28187 raft_consensus.cc:504] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b53502e2a1ae4c86a054d38048dd0df6" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 46775 } } peers { permanent_uuid: "9d313001b0a647caba8f53540a2b67ac" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 46393 } attrs { promote: false } }
I20240503 08:12:33.034905 28187 leader_election.cc:288] T 3c65da71faf34351885b5033d3d68835 P 9d313001b0a647caba8f53540a2b67ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b53502e2a1ae4c86a054d38048dd0df6 (127.20.107.129:46775)
I20240503 08:12:33.035574 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 25879
I20240503 08:12:33.060269 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 26009
I20240503 08:12:33.067014 26133 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:33.226207 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 26009
W20240503 08:12:33.255362 27415 connection.cc:535] server connection from 127.20.107.135:50461 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:12:33.255703 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 27398
I20240503 08:12:33.262890 27521 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:33.389432 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 27398
W20240503 08:12:33.410307 24229 connection.cc:535] server connection from 127.20.107.136:45279 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:12:33.410498 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 24214
I20240503 08:12:33.411604 24274 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:33.577003 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 24214
2024-05-03T08:12:33Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/3 (51678 ms)
[----------] 1 test from RebalanceParamTest (51678 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240503 08:12:33.673169 20910 test_util.cc:268] Using random seed: 1210418113
I20240503 08:12:33.675350 20910 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:12:33.675415 20910 ts_itest-base.cc:113] --------------
I20240503 08:12:33.675436 20910 ts_itest-base.cc:114] 8 tablet servers
I20240503 08:12:33.675454 20910 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:12:33.675472 20910 ts_itest-base.cc:116] --------------
2024-05-03T08:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:12:33Z Disabled control of system clock
I20240503 08:12:33.728616 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.107.190:33183
--webserver_interface=127.20.107.190
--webserver_port=0
--builtin_ntp_servers=127.20.107.148:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.107.190:33183
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 08:12:34.081079 28202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:34.081431 28202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:34.081532 28202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:34.089344 28202 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:34.089486 28202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:12:34.089534 28202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:34.089583 28202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:12:34.089632 28202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:12:34.100050 28202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.107.190:33183
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.107.190:33183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.107.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:34.101430 28202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:34.103137 28202 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:34.114328 28207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:34.114823 28208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:34.115460 28210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:34.116357 28202 server_base.cc:1034] running on GCE node
I20240503 08:12:34.117192 28202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:34.118650 28202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:34.119855 28202 hybrid_clock.cc:648] HybridClock initialized: now 1714723954119763 us; error 85 us; skew 500 ppm
I20240503 08:12:34.120188 28202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:34.127496 28202 webserver.cc:460] Webserver started at http://127.20.107.190:43649/ using document root <none> and password file <none>
I20240503 08:12:34.128309 28202 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:34.128448 28202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:34.128787 28202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:34.131604 28202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e8f5acc8744f4dccbfd654a6d7dab5bf"
format_stamp: "Formatted at 2024-05-03 08:12:34 on dist-test-slave-4tl1"
I20240503 08:12:34.132436 28202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e8f5acc8744f4dccbfd654a6d7dab5bf"
format_stamp: "Formatted at 2024-05-03 08:12:34 on dist-test-slave-4tl1"
I20240503 08:12:34.138010 28202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20240503 08:12:34.141505 28216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:34.142971 28202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:12:34.143154 28202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e8f5acc8744f4dccbfd654a6d7dab5bf"
format_stamp: "Formatted at 2024-05-03 08:12:34 on dist-test-slave-4tl1"
I20240503 08:12:34.143363 28202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:34.175769 28202 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:34.176712 28202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:34.177023 28202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:34.197211 28267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.190:33183 every 8 connection(s)
I20240503 08:12:34.197206 28202 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.190:33183
I20240503 08:12:34.198823 28202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:12:34.200971 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 28202
I20240503 08:12:34.201170 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:12:34.203722 28268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.215293 28268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf: Bootstrap starting.
I20240503 08:12:34.218317 28268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.219291 28268 log.cc:824] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:34.221976 28268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf: No bootstrap required, opened a new log
I20240503 08:12:34.228097 28268 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f5acc8744f4dccbfd654a6d7dab5bf" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 33183 } }
I20240503 08:12:34.228425 28268 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.228504 28268 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f5acc8744f4dccbfd654a6d7dab5bf, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.229082 28268 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f5acc8744f4dccbfd654a6d7dab5bf" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 33183 } }
I20240503 08:12:34.229274 28268 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:12:34.229367 28268 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:12:34.229486 28268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.231491 28268 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f5acc8744f4dccbfd654a6d7dab5bf" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 33183 } }
I20240503 08:12:34.231948 28268 leader_election.cc:302] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f5acc8744f4dccbfd654a6d7dab5bf; no voters:
I20240503 08:12:34.232491 28268 leader_election.cc:288] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:12:34.232628 28273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:34.233559 28273 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [term 1 LEADER]: Becoming Leader. State: Replica: e8f5acc8744f4dccbfd654a6d7dab5bf, State: Running, Role: LEADER
I20240503 08:12:34.234120 28273 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f5acc8744f4dccbfd654a6d7dab5bf" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 33183 } }
I20240503 08:12:34.234990 28268 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:12:34.237017 28275 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8f5acc8744f4dccbfd654a6d7dab5bf. Latest consensus state: current_term: 1 leader_uuid: "e8f5acc8744f4dccbfd654a6d7dab5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f5acc8744f4dccbfd654a6d7dab5bf" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 33183 } } }
I20240503 08:12:34.237341 28275 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:34.237125 28274 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8f5acc8744f4dccbfd654a6d7dab5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f5acc8744f4dccbfd654a6d7dab5bf" member_type: VOTER last_known_addr { host: "127.20.107.190" port: 33183 } } }
I20240503 08:12:34.237509 28274 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:34.238950 28278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:12:34.244444 28278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:12:34.254225 28278 catalog_manager.cc:1347] Generated new cluster ID: c8ed4da5d98d43e0bd279e561db2373a
I20240503 08:12:34.254374 28278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:12:34.279127 28278 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:12:34.280990 28278 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:12:34.292322 28278 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e8f5acc8744f4dccbfd654a6d7dab5bf: Generated new TSK 0
I20240503 08:12:34.293275 28278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:12:34.299226 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.129:0
--local_ip_for_outbound_sockets=127.20.107.129
--webserver_interface=127.20.107.129
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--builtin_ntp_servers=127.20.107.148:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:12:34.669348 28292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:34.669665 28292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:34.669776 28292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:34.677889 28292 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:34.678036 28292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:34.678200 28292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.129
I20240503 08:12:34.687835 28292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.107.129
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.129
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:34.689281 28292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:34.691045 28292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:34.703725 28297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:34.704515 28298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:34.704982 28300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:34.705842 28292 server_base.cc:1034] running on GCE node
I20240503 08:12:34.706523 28292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:34.707471 28292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:34.708675 28292 hybrid_clock.cc:648] HybridClock initialized: now 1714723954708626 us; error 45 us; skew 500 ppm
I20240503 08:12:34.709038 28292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:34.711498 28292 webserver.cc:460] Webserver started at http://127.20.107.129:38733/ using document root <none> and password file <none>
I20240503 08:12:34.712210 28292 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:34.712316 28292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:34.712600 28292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:34.717463 28292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
format_stamp: "Formatted at 2024-05-03 08:12:34 on dist-test-slave-4tl1"
I20240503 08:12:34.718300 28292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
format_stamp: "Formatted at 2024-05-03 08:12:34 on dist-test-slave-4tl1"
I20240503 08:12:34.724129 28292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20240503 08:12:34.728013 28306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:34.729729 28292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:34.729908 28292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
format_stamp: "Formatted at 2024-05-03 08:12:34 on dist-test-slave-4tl1"
I20240503 08:12:34.730131 28292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:34.753181 28292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:34.754097 28292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:34.754377 28292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:34.755429 28292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:34.757035 28292 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:34.757126 28292 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:34.757238 28292 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:34.757326 28292 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:34.792268 28292 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.129:38529
I20240503 08:12:34.792316 28418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.129:38529 every 8 connection(s)
I20240503 08:12:34.793709 28292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:12:34.801343 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 28292
I20240503 08:12:34.801512 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:12:34.805269 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.130:0
--local_ip_for_outbound_sockets=127.20.107.130
--webserver_interface=127.20.107.130
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--builtin_ntp_servers=127.20.107.148:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:12:34.808672 28419 heartbeater.cc:347] Connected to a master server at 127.20.107.190:33183
I20240503 08:12:34.809026 28419 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:34.809917 28419 heartbeater.cc:511] Master 127.20.107.190:33183 requested a full tablet report, sending...
I20240503 08:12:34.812099 28233 ts_manager.cc:194] Registered new tserver with Master: 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:12:34.813637 28233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.129:43949
W20240503 08:12:35.158020 28423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:35.158505 28423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:35.158689 28423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:35.168330 28423 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:35.168493 28423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:35.168610 28423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.130
I20240503 08:12:35.178414 28423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.107.130
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.130
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:35.179838 28423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:35.181740 28423 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:35.194890 28428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:35.195660 28431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:35.196125 28429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:35.197167 28423 server_base.cc:1034] running on GCE node
I20240503 08:12:35.198104 28423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:35.199347 28423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:35.200610 28423 hybrid_clock.cc:648] HybridClock initialized: now 1714723955200549 us; error 57 us; skew 500 ppm
I20240503 08:12:35.201045 28423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:35.203977 28423 webserver.cc:460] Webserver started at http://127.20.107.130:38941/ using document root <none> and password file <none>
I20240503 08:12:35.204946 28423 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:35.205082 28423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:35.205464 28423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:35.209012 28423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
format_stamp: "Formatted at 2024-05-03 08:12:35 on dist-test-slave-4tl1"
I20240503 08:12:35.209990 28423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
format_stamp: "Formatted at 2024-05-03 08:12:35 on dist-test-slave-4tl1"
I20240503 08:12:35.216667 28423 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240503 08:12:35.220403 28437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:35.221823 28423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:35.221987 28423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
format_stamp: "Formatted at 2024-05-03 08:12:35 on dist-test-slave-4tl1"
I20240503 08:12:35.222180 28423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:35.239878 28423 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:35.240769 28423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:35.241057 28423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:35.241986 28423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:35.243454 28423 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:35.243549 28423 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:35.243665 28423 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:35.243746 28423 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:35.281256 28423 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.130:40397
I20240503 08:12:35.281312 28549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.130:40397 every 8 connection(s)
I20240503 08:12:35.282604 28423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:12:35.287871 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 28423
I20240503 08:12:35.288017 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:12:35.291006 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.131:0
--local_ip_for_outbound_sockets=127.20.107.131
--webserver_interface=127.20.107.131
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--builtin_ntp_servers=127.20.107.148:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:12:35.296047 28550 heartbeater.cc:347] Connected to a master server at 127.20.107.190:33183
I20240503 08:12:35.296376 28550 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:35.297276 28550 heartbeater.cc:511] Master 127.20.107.190:33183 requested a full tablet report, sending...
I20240503 08:12:35.298805 28233 ts_manager.cc:194] Registered new tserver with Master: 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:35.299923 28233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.130:45561
W20240503 08:12:35.640707 28553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:35.641110 28553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:35.641225 28553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:35.648914 28553 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:35.649065 28553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:35.649191 28553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.131
I20240503 08:12:35.658504 28553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.107.131
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.131
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:35.659865 28553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:35.661657 28553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:35.675256 28559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:35.675951 28558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:35.676666 28561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:35.677584 28553 server_base.cc:1034] running on GCE node
I20240503 08:12:35.678273 28553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:35.679292 28553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:35.680545 28553 hybrid_clock.cc:648] HybridClock initialized: now 1714723955680493 us; error 41 us; skew 500 ppm
I20240503 08:12:35.680950 28553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:35.683460 28553 webserver.cc:460] Webserver started at http://127.20.107.131:37507/ using document root <none> and password file <none>
I20240503 08:12:35.684247 28553 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:35.684381 28553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:35.684716 28553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:35.687515 28553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4574396f35514c9caccb346f1a7af84d"
format_stamp: "Formatted at 2024-05-03 08:12:35 on dist-test-slave-4tl1"
I20240503 08:12:35.688324 28553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4574396f35514c9caccb346f1a7af84d"
format_stamp: "Formatted at 2024-05-03 08:12:35 on dist-test-slave-4tl1"
I20240503 08:12:35.693725 28553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240503 08:12:35.697444 28567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:35.698865 28553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 08:12:35.699148 28553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4574396f35514c9caccb346f1a7af84d"
format_stamp: "Formatted at 2024-05-03 08:12:35 on dist-test-slave-4tl1"
I20240503 08:12:35.699359 28553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:35.719847 28553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:35.720837 28553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:35.721225 28553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:35.722453 28553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:35.724051 28553 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:35.724162 28553 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:35.724282 28553 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:35.724371 28553 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:35.758949 28553 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.131:44559
I20240503 08:12:35.759006 28679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.131:44559 every 8 connection(s)
I20240503 08:12:35.760283 28553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:12:35.763777 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 28553
I20240503 08:12:35.763962 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:12:35.767832 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.132:0
--local_ip_for_outbound_sockets=127.20.107.132
--webserver_interface=127.20.107.132
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--builtin_ntp_servers=127.20.107.148:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:12:35.775579 28680 heartbeater.cc:347] Connected to a master server at 127.20.107.190:33183
I20240503 08:12:35.775923 28680 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:35.776904 28680 heartbeater.cc:511] Master 127.20.107.190:33183 requested a full tablet report, sending...
I20240503 08:12:35.778713 28233 ts_manager.cc:194] Registered new tserver with Master: 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:12:35.779618 28233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.131:47123
I20240503 08:12:35.817028 28419 heartbeater.cc:503] Master 127.20.107.190:33183 was elected leader, sending a full tablet report...
W20240503 08:12:36.124353 28683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:36.124672 28683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:36.124778 28683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:36.132369 28683 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:36.132505 28683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:36.132661 28683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.132
I20240503 08:12:36.142351 28683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.107.132
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.132
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:36.143712 28683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:36.145547 28683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:36.159482 28688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:36.159826 28689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:36.160642 28691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:36.161604 28683 server_base.cc:1034] running on GCE node
I20240503 08:12:36.162444 28683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:36.163803 28683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:36.165113 28683 hybrid_clock.cc:648] HybridClock initialized: now 1714723956165049 us; error 57 us; skew 500 ppm
I20240503 08:12:36.166056 28683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:36.169335 28683 webserver.cc:460] Webserver started at http://127.20.107.132:43459/ using document root <none> and password file <none>
I20240503 08:12:36.170135 28683 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:36.170248 28683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:36.170590 28683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:36.173463 28683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "93f7e733cec9415eb422b0a7ba8621cb"
format_stamp: "Formatted at 2024-05-03 08:12:36 on dist-test-slave-4tl1"
I20240503 08:12:36.174278 28683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "93f7e733cec9415eb422b0a7ba8621cb"
format_stamp: "Formatted at 2024-05-03 08:12:36 on dist-test-slave-4tl1"
I20240503 08:12:36.179744 28683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:12:36.184154 28697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:36.185904 28683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 08:12:36.186084 28683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "93f7e733cec9415eb422b0a7ba8621cb"
format_stamp: "Formatted at 2024-05-03 08:12:36 on dist-test-slave-4tl1"
I20240503 08:12:36.186300 28683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:36.202755 28683 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:36.203619 28683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:36.203912 28683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:36.204890 28683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:36.206737 28683 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:36.206833 28683 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:36.206912 28683 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:36.206967 28683 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:36.241413 28683 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.132:34095
I20240503 08:12:36.241461 28809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.132:34095 every 8 connection(s)
I20240503 08:12:36.242966 28683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:12:36.252658 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 28683
I20240503 08:12:36.252815 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:12:36.256462 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.133:0
--local_ip_for_outbound_sockets=127.20.107.133
--webserver_interface=127.20.107.133
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--builtin_ntp_servers=127.20.107.148:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:12:36.256507 28810 heartbeater.cc:347] Connected to a master server at 127.20.107.190:33183
I20240503 08:12:36.256860 28810 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:36.258229 28810 heartbeater.cc:511] Master 127.20.107.190:33183 requested a full tablet report, sending...
I20240503 08:12:36.260589 28233 ts_manager.cc:194] Registered new tserver with Master: 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:36.261816 28233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.132:38521
I20240503 08:12:36.303128 28550 heartbeater.cc:503] Master 127.20.107.190:33183 was elected leader, sending a full tablet report...
W20240503 08:12:36.620397 28813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:36.620747 28813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:36.620901 28813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:36.628671 28813 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:36.628841 28813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:36.629029 28813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.133
I20240503 08:12:36.638382 28813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.107.133
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.133
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:36.639659 28813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:36.641506 28813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:36.653870 28818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:36.654621 28821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:36.655229 28819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:36.656361 28813 server_base.cc:1034] running on GCE node
I20240503 08:12:36.657063 28813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:36.658068 28813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:36.659278 28813 hybrid_clock.cc:648] HybridClock initialized: now 1714723956659243 us; error 36 us; skew 500 ppm
I20240503 08:12:36.659606 28813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:36.661962 28813 webserver.cc:460] Webserver started at http://127.20.107.133:46727/ using document root <none> and password file <none>
I20240503 08:12:36.662678 28813 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:36.662802 28813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:36.663121 28813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:36.665957 28813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c8375219addc48fa99ac474ed1c9aef9"
format_stamp: "Formatted at 2024-05-03 08:12:36 on dist-test-slave-4tl1"
I20240503 08:12:36.666752 28813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c8375219addc48fa99ac474ed1c9aef9"
format_stamp: "Formatted at 2024-05-03 08:12:36 on dist-test-slave-4tl1"
I20240503 08:12:36.672364 28813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20240503 08:12:36.676187 28827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:36.677762 28813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:36.677960 28813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c8375219addc48fa99ac474ed1c9aef9"
format_stamp: "Formatted at 2024-05-03 08:12:36 on dist-test-slave-4tl1"
I20240503 08:12:36.678169 28813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:36.694725 28813 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:36.695590 28813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:36.695874 28813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:36.696908 28813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:36.698457 28813 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:36.698544 28813 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:36.698617 28813 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:36.698674 28813 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:36.732369 28813 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.133:35121
I20240503 08:12:36.732467 28939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.133:35121 every 8 connection(s)
I20240503 08:12:36.733745 28813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:12:36.736091 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 28813
I20240503 08:12:36.736274 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:12:36.739619 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.134:0
--local_ip_for_outbound_sockets=127.20.107.134
--webserver_interface=127.20.107.134
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--builtin_ntp_servers=127.20.107.148:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:12:36.747833 28940 heartbeater.cc:347] Connected to a master server at 127.20.107.190:33183
I20240503 08:12:36.748142 28940 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:36.749097 28940 heartbeater.cc:511] Master 127.20.107.190:33183 requested a full tablet report, sending...
I20240503 08:12:36.750509 28233 ts_manager.cc:194] Registered new tserver with Master: c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121)
I20240503 08:12:36.751257 28233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.133:59011
I20240503 08:12:36.783023 28680 heartbeater.cc:503] Master 127.20.107.190:33183 was elected leader, sending a full tablet report...
W20240503 08:12:37.090579 28943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:37.090893 28943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:37.091017 28943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:37.098572 28943 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:37.098722 28943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:37.098937 28943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.134
I20240503 08:12:37.109277 28943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.107.134
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.134
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:37.110610 28943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:37.112452 28943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:37.124981 28951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:37.125576 28948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:37.126175 28949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:37.127022 28943 server_base.cc:1034] running on GCE node
I20240503 08:12:37.127641 28943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:37.128559 28943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:37.129765 28943 hybrid_clock.cc:648] HybridClock initialized: now 1714723957129706 us; error 50 us; skew 500 ppm
I20240503 08:12:37.130100 28943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:37.132459 28943 webserver.cc:460] Webserver started at http://127.20.107.134:34649/ using document root <none> and password file <none>
I20240503 08:12:37.133222 28943 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:37.133322 28943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:37.133653 28943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:37.136462 28943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "21d21a5b596b4d48a17a1fb54a972844"
format_stamp: "Formatted at 2024-05-03 08:12:37 on dist-test-slave-4tl1"
I20240503 08:12:37.137229 28943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "21d21a5b596b4d48a17a1fb54a972844"
format_stamp: "Formatted at 2024-05-03 08:12:37 on dist-test-slave-4tl1"
I20240503 08:12:37.142534 28943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 08:12:37.146186 28957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:37.147493 28943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240503 08:12:37.147676 28943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "21d21a5b596b4d48a17a1fb54a972844"
format_stamp: "Formatted at 2024-05-03 08:12:37 on dist-test-slave-4tl1"
I20240503 08:12:37.147878 28943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:37.170580 28943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:37.171553 28943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:37.171995 28943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:37.173480 28943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:37.175236 28943 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:37.175366 28943 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:37.175489 28943 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:37.175575 28943 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:37.208395 28943 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.134:34587
I20240503 08:12:37.208429 29069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.134:34587 every 8 connection(s)
I20240503 08:12:37.210016 28943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:12:37.215435 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 28943
I20240503 08:12:37.215600 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:12:37.219081 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.135:0
--local_ip_for_outbound_sockets=127.20.107.135
--webserver_interface=127.20.107.135
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--builtin_ntp_servers=127.20.107.148:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:12:37.223400 29070 heartbeater.cc:347] Connected to a master server at 127.20.107.190:33183
I20240503 08:12:37.223701 29070 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:37.224541 29070 heartbeater.cc:511] Master 127.20.107.190:33183 requested a full tablet report, sending...
I20240503 08:12:37.225953 28233 ts_manager.cc:194] Registered new tserver with Master: 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
I20240503 08:12:37.226773 28233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.134:37205
I20240503 08:12:37.269069 28810 heartbeater.cc:503] Master 127.20.107.190:33183 was elected leader, sending a full tablet report...
W20240503 08:12:37.578027 29073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:37.578349 29073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:37.578473 29073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:37.586503 29073 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:37.586664 29073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:37.586833 29073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.135
I20240503 08:12:37.596151 29073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.107.135
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.135
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:37.597503 29073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:37.599207 29073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:37.612347 29081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:37.613332 29078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:37.613896 29079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:37.615329 29073 server_base.cc:1034] running on GCE node
I20240503 08:12:37.615978 29073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:37.617058 29073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:37.618312 29073 hybrid_clock.cc:648] HybridClock initialized: now 1714723957618240 us; error 52 us; skew 500 ppm
I20240503 08:12:37.618711 29073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:37.621273 29073 webserver.cc:460] Webserver started at http://127.20.107.135:38853/ using document root <none> and password file <none>
I20240503 08:12:37.622035 29073 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:37.622162 29073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:37.622715 29073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:37.625552 29073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "bea2145b8e24436392b18ec3cfe90b93"
format_stamp: "Formatted at 2024-05-03 08:12:37 on dist-test-slave-4tl1"
I20240503 08:12:37.626346 29073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "bea2145b8e24436392b18ec3cfe90b93"
format_stamp: "Formatted at 2024-05-03 08:12:37 on dist-test-slave-4tl1"
I20240503 08:12:37.631748 29073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:12:37.635351 29087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:37.636828 29073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:37.637050 29073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "bea2145b8e24436392b18ec3cfe90b93"
format_stamp: "Formatted at 2024-05-03 08:12:37 on dist-test-slave-4tl1"
I20240503 08:12:37.637253 29073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:37.658677 29073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:37.659718 29073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:37.660001 29073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:37.661216 29073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:37.662760 29073 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:37.662865 29073 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:37.662981 29073 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:37.663053 29073 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:37.696319 29073 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.135:45937
I20240503 08:12:37.696367 29199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.135:45937 every 8 connection(s)
I20240503 08:12:37.697685 29073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:12:37.702040 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 29073
I20240503 08:12:37.702188 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:12:37.705289 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.136:0
--local_ip_for_outbound_sockets=127.20.107.136
--webserver_interface=127.20.107.136
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--builtin_ntp_servers=127.20.107.148:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:12:37.711143 29200 heartbeater.cc:347] Connected to a master server at 127.20.107.190:33183
I20240503 08:12:37.711473 29200 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:37.712345 29200 heartbeater.cc:511] Master 127.20.107.190:33183 requested a full tablet report, sending...
I20240503 08:12:37.713878 28233 ts_manager.cc:194] Registered new tserver with Master: bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937)
I20240503 08:12:37.714982 28233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.135:49615
I20240503 08:12:37.754628 28940 heartbeater.cc:503] Master 127.20.107.190:33183 was elected leader, sending a full tablet report...
W20240503 08:12:38.059690 29203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:38.059993 29203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:38.060084 29203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:38.067620 29203 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:38.067770 29203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:38.067888 29203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.136
I20240503 08:12:38.077493 29203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.107.136
--webserver_port=0
--tserver_master_addrs=127.20.107.190:33183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.136
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:38.078871 29203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:38.080611 29203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:38.093247 29208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:38.093899 29209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:38.094491 29211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:38.095361 29203 server_base.cc:1034] running on GCE node
I20240503 08:12:38.095971 29203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:38.097039 29203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:38.098275 29203 hybrid_clock.cc:648] HybridClock initialized: now 1714723958098202 us; error 61 us; skew 500 ppm
I20240503 08:12:38.098646 29203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:38.101162 29203 webserver.cc:460] Webserver started at http://127.20.107.136:37883/ using document root <none> and password file <none>
I20240503 08:12:38.101924 29203 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:38.102030 29203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:38.102319 29203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:38.105218 29203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "b29361eb4afa4ab3af7d8019d8a5042d"
format_stamp: "Formatted at 2024-05-03 08:12:38 on dist-test-slave-4tl1"
I20240503 08:12:38.106314 29203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "b29361eb4afa4ab3af7d8019d8a5042d"
format_stamp: "Formatted at 2024-05-03 08:12:38 on dist-test-slave-4tl1"
I20240503 08:12:38.112501 29203 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20240503 08:12:38.116400 29217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:38.118013 29203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:38.118197 29203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "b29361eb4afa4ab3af7d8019d8a5042d"
format_stamp: "Formatted at 2024-05-03 08:12:38 on dist-test-slave-4tl1"
I20240503 08:12:38.118412 29203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:38.138424 29203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:38.139284 29203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:38.139551 29203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:38.140506 29203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:38.142089 29203 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:38.142200 29203 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:38.142320 29203 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:38.142397 29203 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:38.177512 29203 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.136:42389
I20240503 08:12:38.177565 29329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.136:42389 every 8 connection(s)
I20240503 08:12:38.178804 29203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:12:38.188679 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 29203
I20240503 08:12:38.189358 20910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:12:38.191819 29330 heartbeater.cc:347] Connected to a master server at 127.20.107.190:33183
I20240503 08:12:38.192132 29330 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:38.193089 29330 heartbeater.cc:511] Master 127.20.107.190:33183 requested a full tablet report, sending...
I20240503 08:12:38.194445 28233 ts_manager.cc:194] Registered new tserver with Master: b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389)
I20240503 08:12:38.195235 28233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.136:41761
I20240503 08:12:38.205821 20910 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:12:38.223971 20910 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:12:38.224117 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 28813
I20240503 08:12:38.229254 29070 heartbeater.cc:503] Master 127.20.107.190:33183 was elected leader, sending a full tablet report...
I20240503 08:12:38.233351 28935 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:38.320523 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 28813
I20240503 08:12:38.337724 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 28943
I20240503 08:12:38.346992 29065 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:38.430220 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 28943
I20240503 08:12:38.446565 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 29073
I20240503 08:12:38.454618 29195 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:38.538743 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 29073
I20240503 08:12:38.555490 20910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu pid 29203
I20240503 08:12:38.564399 29325 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:38.646596 20910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu with pid 29203
I20240503 08:12:42.414781 28233 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60904:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:42.454032 28354 tablet_service.cc:1467] Processing CreateTablet for tablet ede24a27608a4cf196dd46f85c4163e1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2cf824bda7e41fb9ad499d46ed7c4d1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.454032 28745 tablet_service.cc:1467] Processing CreateTablet for tablet ede24a27608a4cf196dd46f85c4163e1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2cf824bda7e41fb9ad499d46ed7c4d1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.454030 28615 tablet_service.cc:1467] Processing CreateTablet for tablet 745d736d43244d4f9063f61de77cb473 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2cf824bda7e41fb9ad499d46ed7c4d1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.455286 28744 tablet_service.cc:1467] Processing CreateTablet for tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2cf824bda7e41fb9ad499d46ed7c4d1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.455610 28615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745d736d43244d4f9063f61de77cb473. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.455614 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede24a27608a4cf196dd46f85c4163e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.456035 28744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.457427 28614 tablet_service.cc:1467] Processing CreateTablet for tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2cf824bda7e41fb9ad499d46ed7c4d1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.457964 28614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.461880 28354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede24a27608a4cf196dd46f85c4163e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.475189 28353 tablet_service.cc:1467] Processing CreateTablet for tablet 745d736d43244d4f9063f61de77cb473 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2cf824bda7e41fb9ad499d46ed7c4d1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.475900 28353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745d736d43244d4f9063f61de77cb473. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.477377 28352 tablet_service.cc:1467] Processing CreateTablet for tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2cf824bda7e41fb9ad499d46ed7c4d1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.477803 28352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.479290 28484 tablet_service.cc:1467] Processing CreateTablet for tablet 745d736d43244d4f9063f61de77cb473 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2cf824bda7e41fb9ad499d46ed7c4d1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.480755 28484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745d736d43244d4f9063f61de77cb473. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.481571 28485 tablet_service.cc:1467] Processing CreateTablet for tablet ede24a27608a4cf196dd46f85c4163e1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2cf824bda7e41fb9ad499d46ed7c4d1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.482053 28485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede24a27608a4cf196dd46f85c4163e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.497195 29359 tablet_bootstrap.cc:492] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:42.498468 29360 tablet_bootstrap.cc:492] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:42.499303 29361 tablet_bootstrap.cc:492] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:42.500824 29362 tablet_bootstrap.cc:492] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:42.502573 29360 tablet_bootstrap.cc:654] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.502573 29361 tablet_bootstrap.cc:654] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.503302 29359 tablet_bootstrap.cc:654] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.504287 29362 tablet_bootstrap.cc:654] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.504366 29360 log.cc:824] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:42.505023 29361 log.cc:824] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:42.505120 29359 log.cc:824] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:42.505659 29362 log.cc:824] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:42.510180 29362 tablet_bootstrap.cc:492] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:42.510666 29362 ts_tablet_manager.cc:1367] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 08:12:42.519630 29361 tablet_bootstrap.cc:492] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:42.519794 29360 tablet_bootstrap.cc:492] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:42.519966 29361 ts_tablet_manager.cc:1367] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.004s
I20240503 08:12:42.520197 29360 ts_tablet_manager.cc:1367] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.008s
I20240503 08:12:42.521644 29359 tablet_bootstrap.cc:492] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:42.521968 29359 ts_tablet_manager.cc:1367] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.011s
I20240503 08:12:42.522855 29362 raft_consensus.cc:348] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.523428 29362 raft_consensus.cc:374] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.523581 29362 raft_consensus.cc:729] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.524513 29362 consensus_queue.cc:260] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.529484 29361 raft_consensus.cc:348] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.530004 29361 raft_consensus.cc:374] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.530140 29361 raft_consensus.cc:729] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.530756 29359 raft_consensus.cc:348] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:42.530908 29361 consensus_queue.cc:260] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.531222 29359 raft_consensus.cc:374] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.531431 29359 raft_consensus.cc:729] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.531286 29360 raft_consensus.cc:348] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.531917 29360 raft_consensus.cc:374] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.532100 29360 raft_consensus.cc:729] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.533120 29360 consensus_queue.cc:260] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.534267 29359 consensus_queue.cc:260] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:42.537600 29360 ts_tablet_manager.cc:1398] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.017s user 0.003s sys 0.011s
I20240503 08:12:42.540359 29362 ts_tablet_manager.cc:1398] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.029s user 0.015s sys 0.007s
I20240503 08:12:42.542399 29360 tablet_bootstrap.cc:492] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:42.543118 29362 tablet_bootstrap.cc:492] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:42.544399 29360 tablet_bootstrap.cc:654] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.544790 29362 tablet_bootstrap.cc:654] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.547151 29361 ts_tablet_manager.cc:1398] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.027s user 0.014s sys 0.011s
I20240503 08:12:42.547662 29361 tablet_bootstrap.cc:492] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:42.547981 29359 ts_tablet_manager.cc:1398] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.026s user 0.016s sys 0.007s
I20240503 08:12:42.548468 29359 tablet_bootstrap.cc:492] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:42.549413 29361 tablet_bootstrap.cc:654] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.550179 29359 tablet_bootstrap.cc:654] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
W20240503 08:12:42.551877 28420 tablet.cc:2369] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:42.556838 29360 tablet_bootstrap.cc:492] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:42.557607 29360 ts_tablet_manager.cc:1367] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20240503 08:12:42.559146 29360 raft_consensus.cc:348] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:42.559545 29360 raft_consensus.cc:374] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.559703 29360 raft_consensus.cc:729] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.560103 29360 consensus_queue.cc:260] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:42.563892 29360 ts_tablet_manager.cc:1398] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:12:42.568243 29362 tablet_bootstrap.cc:492] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:42.568418 29362 ts_tablet_manager.cc:1367] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.016s
I20240503 08:12:42.568754 29361 tablet_bootstrap.cc:492] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:42.568969 29361 ts_tablet_manager.cc:1367] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.012s
I20240503 08:12:42.569365 29359 tablet_bootstrap.cc:492] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:42.569528 29359 ts_tablet_manager.cc:1367] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.005s
I20240503 08:12:42.569416 29362 raft_consensus.cc:348] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.569617 29362 raft_consensus.cc:374] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.569739 29362 raft_consensus.cc:729] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.569977 29362 consensus_queue.cc:260] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.569963 29361 raft_consensus.cc:348] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.570163 29361 raft_consensus.cc:374] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.570276 29361 raft_consensus.cc:729] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.570509 29361 consensus_queue.cc:260] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.570501 29359 raft_consensus.cc:348] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.570699 29359 raft_consensus.cc:374] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.570849 29359 raft_consensus.cc:729] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.571193 29361 ts_tablet_manager.cc:1398] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:42.571090 29359 consensus_queue.cc:260] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.571656 29359 ts_tablet_manager.cc:1398] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:42.571925 29362 ts_tablet_manager.cc:1398] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:42.572250 29362 tablet_bootstrap.cc:492] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:42.574129 29362 tablet_bootstrap.cc:654] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.576292 29362 tablet_bootstrap.cc:492] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:42.576433 29362 ts_tablet_manager.cc:1367] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:42.577042 29362 raft_consensus.cc:348] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:42.577227 29362 raft_consensus.cc:374] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.577282 29362 raft_consensus.cc:729] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.577492 29362 consensus_queue.cc:260] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:42.577947 29362 ts_tablet_manager.cc:1398] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:42.592201 29369 raft_consensus.cc:482] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:42.592500 29369 raft_consensus.cc:504] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:42.594444 29369 leader_election.cc:288] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:12:42.601773 28635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfa8537bc8a4950b53fbf3c36d5ecf3" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
I20240503 08:12:42.601836 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfa8537bc8a4950b53fbf3c36d5ecf3" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" is_pre_election: true
I20240503 08:12:42.602274 28635 raft_consensus.cc:2454] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 0.
I20240503 08:12:42.602300 28374 raft_consensus.cc:2454] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 0.
I20240503 08:12:42.602885 28699 leader_election.cc:302] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:42.603266 29369 raft_consensus.cc:2781] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:42.603415 29369 raft_consensus.cc:482] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:42.603530 29369 raft_consensus.cc:3037] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.605669 29369 raft_consensus.cc:504] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:42.606336 29369 leader_election.cc:288] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 election: Requested vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:12:42.606725 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfa8537bc8a4950b53fbf3c36d5ecf3" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:12:42.606786 28635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfa8537bc8a4950b53fbf3c36d5ecf3" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:42.606914 28374 raft_consensus.cc:3037] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.606984 28635 raft_consensus.cc:3037] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.609010 28374 raft_consensus.cc:2454] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 1.
I20240503 08:12:42.609058 28635 raft_consensus.cc:2454] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 1.
I20240503 08:12:42.609416 28699 leader_election.cc:302] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:42.609697 29369 raft_consensus.cc:2781] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:42.610170 29369 raft_consensus.cc:686] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Becoming Leader. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Running, Role: LEADER
I20240503 08:12:42.610675 29369 consensus_queue.cc:237] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:42.615111 28231 catalog_manager.cc:5484] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132). New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:42.647143 29367 raft_consensus.cc:482] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:42.647362 29367 raft_consensus.cc:504] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.649142 29367 leader_election.cc:288] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:42.657652 28505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede24a27608a4cf196dd46f85c4163e1" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:42.657624 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede24a27608a4cf196dd46f85c4163e1" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:42.658115 28765 raft_consensus.cc:2454] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:42.658222 28505 raft_consensus.cc:2454] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:42.658676 28308 leader_election.cc:302] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:42.659083 29367 raft_consensus.cc:2781] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:42.659217 29367 raft_consensus.cc:482] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:42.659319 29367 raft_consensus.cc:3037] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.661350 29367 raft_consensus.cc:504] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.661962 29367 leader_election.cc:288] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Requested vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:42.662446 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede24a27608a4cf196dd46f85c4163e1" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:42.662479 28505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede24a27608a4cf196dd46f85c4163e1" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:12:42.662636 28765 raft_consensus.cc:3037] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.662654 28505 raft_consensus.cc:3037] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.664630 28765 raft_consensus.cc:2454] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:42.664673 28505 raft_consensus.cc:2454] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:42.665196 28308 leader_election.cc:302] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:42.665633 29367 raft_consensus.cc:2781] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:42.666456 29367 raft_consensus.cc:686] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Becoming Leader. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Running, Role: LEADER
I20240503 08:12:42.667032 29367 consensus_queue.cc:237] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.671835 28231 catalog_manager.cc:5484] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129). New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:42.749594 28811 tablet.cc:2369] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:12:42.767856 28681 tablet.cc:2369] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:42.786857 29368 raft_consensus.cc:482] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:42.787112 29368 raft_consensus.cc:504] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.788763 29368 leader_election.cc:288] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
W20240503 08:12:42.790269 28551 tablet.cc:2369] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:42.796842 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745d736d43244d4f9063f61de77cb473" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" is_pre_election: true
I20240503 08:12:42.797165 28374 raft_consensus.cc:2454] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:42.797129 28635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745d736d43244d4f9063f61de77cb473" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
I20240503 08:12:42.797386 28635 raft_consensus.cc:2454] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:42.797566 29370 raft_consensus.cc:482] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:42.797760 28439 leader_election.cc:302] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:42.797761 29370 raft_consensus.cc:504] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.798442 29368 raft_consensus.cc:2781] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:42.798575 29368 raft_consensus.cc:482] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:42.798676 29368 raft_consensus.cc:3037] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.799636 29370 leader_election.cc:288] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:42.801162 29368 raft_consensus.cc:504] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.801790 29368 leader_election.cc:288] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Requested vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:12:42.802104 28635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745d736d43244d4f9063f61de77cb473" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:42.802356 28635 raft_consensus.cc:3037] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.802923 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745d736d43244d4f9063f61de77cb473" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:12:42.803140 28374 raft_consensus.cc:3037] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.805292 28635 raft_consensus.cc:2454] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:42.805874 28438 leader_election.cc:302] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:42.806695 28374 raft_consensus.cc:2454] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:42.807010 29368 raft_consensus.cc:2781] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:42.810452 29368 raft_consensus.cc:686] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Becoming Leader. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Running, Role: LEADER
I20240503 08:12:42.811093 29368 consensus_queue.cc:237] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:42.816135 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745d736d43244d4f9063f61de77cb473" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" is_pre_election: true
I20240503 08:12:42.816658 28374 raft_consensus.cc:2379] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4574396f35514c9caccb346f1a7af84d in current term 1: Already voted for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in this term.
I20240503 08:12:42.820384 28505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745d736d43244d4f9063f61de77cb473" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:42.821169 28569 leader_election.cc:302] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d; no voters: 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a
I20240503 08:12:42.821655 29370 raft_consensus.cc:2726] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:42.824062 28231 catalog_manager.cc:5484] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: term changed from 0 to 1, leader changed from <none> to 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130). New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:43.061340 29369 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.074582 29369 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.231745 29368 consensus_queue.cc:1035] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.238067 29388 consensus_queue.cc:1035] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:43.254521 29390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:43.254937 29390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:43.266625 29390 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:43.276684 29367 consensus_queue.cc:1035] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.285449 29367 consensus_queue.cc:1035] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 312.149 ms
time per row: 0.156075 ms
I20240503 08:12:43.772393 28231 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60904:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:43.781850 28484 tablet_service.cc:1467] Processing CreateTablet for tablet c6e1b9f104ea471cab7527bd56433327 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.782346 28484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e1b9f104ea471cab7527bd56433327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.783131 28614 tablet_service.cc:1467] Processing CreateTablet for tablet c6e1b9f104ea471cab7527bd56433327 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.783584 28614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e1b9f104ea471cab7527bd56433327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.786080 28485 tablet_service.cc:1467] Processing CreateTablet for tablet 0caf86850b3841ea811b0a6ab19ae330 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.786175 28354 tablet_service.cc:1467] Processing CreateTablet for tablet b2ebb0b8475b4ed9a09711a87867de08 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.786316 28615 tablet_service.cc:1467] Processing CreateTablet for tablet 0caf86850b3841ea811b0a6ab19ae330 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.786464 28485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caf86850b3841ea811b0a6ab19ae330. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.786636 28354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ebb0b8475b4ed9a09711a87867de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.786695 28615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caf86850b3841ea811b0a6ab19ae330. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.790570 28613 tablet_service.cc:1467] Processing CreateTablet for tablet b2ebb0b8475b4ed9a09711a87867de08 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.790642 28483 tablet_service.cc:1467] Processing CreateTablet for tablet b2ebb0b8475b4ed9a09711a87867de08 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.791040 28613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ebb0b8475b4ed9a09711a87867de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.791214 28483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ebb0b8475b4ed9a09711a87867de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.793418 28745 tablet_service.cc:1467] Processing CreateTablet for tablet c6e1b9f104ea471cab7527bd56433327 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.794016 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e1b9f104ea471cab7527bd56433327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.797351 28744 tablet_service.cc:1467] Processing CreateTablet for tablet 0caf86850b3841ea811b0a6ab19ae330 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.797879 28744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caf86850b3841ea811b0a6ab19ae330. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.802721 29458 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:43.805114 29458 tablet_bootstrap.cc:654] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.822623 29460 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:43.825038 29460 tablet_bootstrap.cc:654] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.825217 29456 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:43.826339 29457 tablet_bootstrap.cc:492] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:43.827546 29456 tablet_bootstrap.cc:654] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.829489 29457 tablet_bootstrap.cc:654] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.829998 29458 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:43.830163 29458 ts_tablet_manager.cc:1367] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.013s
I20240503 08:12:43.830453 29460 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:43.830583 29460 ts_tablet_manager.cc:1367] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240503 08:12:43.831061 29458 raft_consensus.cc:348] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:43.831270 29458 raft_consensus.cc:374] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.831395 29458 raft_consensus.cc:729] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.831399 29460 raft_consensus.cc:348] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:43.831637 29460 raft_consensus.cc:374] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.831730 29460 raft_consensus.cc:729] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.831646 29458 consensus_queue.cc:260] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:43.832114 29460 consensus_queue.cc:260] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:43.832803 29460 ts_tablet_manager.cc:1398] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:43.833305 29458 ts_tablet_manager.cc:1398] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:43.833638 29458 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:43.834538 29460 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:43.835625 29458 tablet_bootstrap.cc:654] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.836789 29460 tablet_bootstrap.cc:654] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.837723 29456 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:43.837960 29456 ts_tablet_manager.cc:1367] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240503 08:12:43.838886 29456 raft_consensus.cc:348] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:43.839196 29456 raft_consensus.cc:374] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.839342 29456 raft_consensus.cc:729] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.839651 29456 consensus_queue.cc:260] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:43.845249 29394 raft_consensus.cc:482] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:43.845544 29394 raft_consensus.cc:504] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:43.846776 29394 leader_election.cc:288] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:43.847281 29456 ts_tablet_manager.cc:1398] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240503 08:12:43.848743 29456 tablet_bootstrap.cc:492] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:43.847608 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0caf86850b3841ea811b0a6ab19ae330" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
W20240503 08:12:43.850376 28698 leader_election.cc:341] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:43.850921 29458 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:43.851146 29458 ts_tablet_manager.cc:1367] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.009s
I20240503 08:12:43.852017 29458 raft_consensus.cc:348] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:43.852274 29458 raft_consensus.cc:374] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.852360 29458 raft_consensus.cc:729] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.852669 29458 consensus_queue.cc:260] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:43.853418 29458 ts_tablet_manager.cc:1398] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:43.853703 29456 tablet_bootstrap.cc:654] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.853785 29458 tablet_bootstrap.cc:492] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:43.855659 29458 tablet_bootstrap.cc:654] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.863142 29456 tablet_bootstrap.cc:492] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:43.863368 29456 ts_tablet_manager.cc:1367] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20240503 08:12:43.864243 29456 raft_consensus.cc:348] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:43.864482 29456 raft_consensus.cc:374] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.864553 29456 raft_consensus.cc:729] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.865710 29456 consensus_queue.cc:260] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:43.866619 29460 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:43.866621 29457 tablet_bootstrap.cc:492] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:43.866793 29457 ts_tablet_manager.cc:1367] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.018s
I20240503 08:12:43.866827 29460 ts_tablet_manager.cc:1367] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.002s
I20240503 08:12:43.867659 29457 raft_consensus.cc:348] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:43.867695 29460 raft_consensus.cc:348] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:43.867887 29457 raft_consensus.cc:374] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.867903 29460 raft_consensus.cc:374] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.867974 29457 raft_consensus.cc:729] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.867967 29460 raft_consensus.cc:729] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.868239 29457 consensus_queue.cc:260] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:43.868252 29460 consensus_queue.cc:260] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:43.868808 29457 ts_tablet_manager.cc:1398] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:43.868954 29460 ts_tablet_manager.cc:1398] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:12:43.869553 29456 ts_tablet_manager.cc:1398] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:12:43.869882 29456 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:43.870908 29458 tablet_bootstrap.cc:492] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:43.871047 29458 ts_tablet_manager.cc:1367] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20240503 08:12:43.871649 29456 tablet_bootstrap.cc:654] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.871977 29458 raft_consensus.cc:348] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:43.872210 29458 raft_consensus.cc:374] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.872316 29458 raft_consensus.cc:729] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.872556 29458 consensus_queue.cc:260] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:43.873296 29458 ts_tablet_manager.cc:1398] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:43.876703 29456 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:43.876961 29456 ts_tablet_manager.cc:1367] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240503 08:12:43.877969 29456 raft_consensus.cc:348] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:43.878274 29456 raft_consensus.cc:374] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.878271 28505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0caf86850b3841ea811b0a6ab19ae330" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:43.878412 29456 raft_consensus.cc:729] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.878528 28505 raft_consensus.cc:2454] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 0.
I20240503 08:12:43.878654 29456 consensus_queue.cc:260] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:43.879436 29456 ts_tablet_manager.cc:1398] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:43.879429 28699 leader_election.cc:302] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 716ffbb2a00b4f43b03e5cedcc9d302a, 93f7e733cec9415eb422b0a7ba8621cb; no voters: 4574396f35514c9caccb346f1a7af84d
I20240503 08:12:43.879823 29394 raft_consensus.cc:2781] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:43.879954 29394 raft_consensus.cc:482] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:43.880041 29394 raft_consensus.cc:3037] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.882155 29394 raft_consensus.cc:504] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:43.882679 29394 leader_election.cc:288] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 election: Requested vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:43.883138 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0caf86850b3841ea811b0a6ab19ae330" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:43.883183 28505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0caf86850b3841ea811b0a6ab19ae330" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:12:43.883337 28634 raft_consensus.cc:3037] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.883348 28505 raft_consensus.cc:3037] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.885241 28505 raft_consensus.cc:2454] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 1.
I20240503 08:12:43.885243 28634 raft_consensus.cc:2454] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 1.
I20240503 08:12:43.885622 28698 leader_election.cc:302] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:43.885893 29394 raft_consensus.cc:2781] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:43.886018 29394 raft_consensus.cc:686] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Becoming Leader. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Running, Role: LEADER
I20240503 08:12:43.886260 29394 consensus_queue.cc:237] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:43.888921 28229 catalog_manager.cc:5484] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132). New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:43.910702 29394 raft_consensus.cc:482] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:43.910945 29394 raft_consensus.cc:504] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:43.911623 29394 leader_election.cc:288] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397), 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:12:43.912192 28505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e1b9f104ea471cab7527bd56433327" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:43.912246 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e1b9f104ea471cab7527bd56433327" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
I20240503 08:12:43.912425 28505 raft_consensus.cc:2454] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 0.
I20240503 08:12:43.912461 28634 raft_consensus.cc:2454] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 0.
I20240503 08:12:43.912839 28699 leader_election.cc:302] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716ffbb2a00b4f43b03e5cedcc9d302a, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:43.913167 29394 raft_consensus.cc:2781] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:43.913268 29394 raft_consensus.cc:482] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:43.913357 29394 raft_consensus.cc:3037] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.915192 29394 raft_consensus.cc:504] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:43.915776 29394 leader_election.cc:288] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 election: Requested vote from peers 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397), 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:12:43.916224 28505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e1b9f104ea471cab7527bd56433327" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:12:43.916281 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e1b9f104ea471cab7527bd56433327" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:43.916437 28505 raft_consensus.cc:3037] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.916483 28634 raft_consensus.cc:3037] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.918354 28505 raft_consensus.cc:2454] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 1.
I20240503 08:12:43.918375 28634 raft_consensus.cc:2454] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 1.
I20240503 08:12:43.918778 28698 leader_election.cc:302] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:43.919056 29394 raft_consensus.cc:2781] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:43.919229 29394 raft_consensus.cc:686] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Becoming Leader. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Running, Role: LEADER
I20240503 08:12:43.919467 29394 consensus_queue.cc:237] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:43.922246 28229 catalog_manager.cc:5484] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132). New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:44.004774 29411 raft_consensus.cc:482] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.005040 29411 raft_consensus.cc:504] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:44.006043 29411 leader_election.cc:288] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:44.006649 28505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ebb0b8475b4ed9a09711a87867de08" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:44.006974 28505 raft_consensus.cc:2454] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:44.007404 28308 leader_election.cc:302] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:44.007725 29411 raft_consensus.cc:2781] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.007860 29411 raft_consensus.cc:482] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.008001 29411 raft_consensus.cc:3037] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.010725 29411 raft_consensus.cc:504] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:44.011323 29411 leader_election.cc:288] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Requested vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:44.011951 28505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ebb0b8475b4ed9a09711a87867de08" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:12:44.012214 28505 raft_consensus.cc:3037] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.013734 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ebb0b8475b4ed9a09711a87867de08" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
I20240503 08:12:44.013778 28635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ebb0b8475b4ed9a09711a87867de08" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:44.014001 28634 raft_consensus.cc:2454] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:44.014683 28505 raft_consensus.cc:2454] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:44.015027 28308 leader_election.cc:302] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters: 4574396f35514c9caccb346f1a7af84d
I20240503 08:12:44.015344 29411 raft_consensus.cc:2781] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.015478 29411 raft_consensus.cc:686] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Becoming Leader. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Running, Role: LEADER
I20240503 08:12:44.015733 29411 consensus_queue.cc:237] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:44.018437 28229 catalog_manager.cc:5484] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129). New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:44.320212 29394 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.326404 29394 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.358151 29369 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.363394 29483 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.399374 29411 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.404959 28634 raft_consensus.cc:3037] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.408655 29490 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:44.413796 29476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:44.414162 29476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:44.421731 29476 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 210.318 ms
time per row: 0.105159 ms
I20240503 08:12:44.784489 28225 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60904:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:44.794660 28354 tablet_service.cc:1467] Processing CreateTablet for tablet e9997e8da99c4f45a63bc3db521b33ba (DEFAULT_TABLE table=rebalance_test_table_2 [id=2ff6aeab7e1847f6884750db73f8c428]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.795192 28354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9997e8da99c4f45a63bc3db521b33ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.795562 28614 tablet_service.cc:1467] Processing CreateTablet for tablet d13c33b8712e4c488abb99b73ff42b86 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2ff6aeab7e1847f6884750db73f8c428]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.795773 28483 tablet_service.cc:1467] Processing CreateTablet for tablet e9997e8da99c4f45a63bc3db521b33ba (DEFAULT_TABLE table=rebalance_test_table_2 [id=2ff6aeab7e1847f6884750db73f8c428]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.795925 28614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d13c33b8712e4c488abb99b73ff42b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.796172 28483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9997e8da99c4f45a63bc3db521b33ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.797809 28744 tablet_service.cc:1467] Processing CreateTablet for tablet e9997e8da99c4f45a63bc3db521b33ba (DEFAULT_TABLE table=rebalance_test_table_2 [id=2ff6aeab7e1847f6884750db73f8c428]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.798216 28744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9997e8da99c4f45a63bc3db521b33ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.801249 28745 tablet_service.cc:1467] Processing CreateTablet for tablet d13c33b8712e4c488abb99b73ff42b86 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2ff6aeab7e1847f6884750db73f8c428]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.801628 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d13c33b8712e4c488abb99b73ff42b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.802443 28352 tablet_service.cc:1467] Processing CreateTablet for tablet d13c33b8712e4c488abb99b73ff42b86 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2ff6aeab7e1847f6884750db73f8c428]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.802858 28352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d13c33b8712e4c488abb99b73ff42b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.809939 28484 tablet_service.cc:1467] Processing CreateTablet for tablet ac392ff7965a405f96efc777aca4f0bb (DEFAULT_TABLE table=rebalance_test_table_2 [id=2ff6aeab7e1847f6884750db73f8c428]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.810406 28484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac392ff7965a405f96efc777aca4f0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.813937 28353 tablet_service.cc:1467] Processing CreateTablet for tablet ac392ff7965a405f96efc777aca4f0bb (DEFAULT_TABLE table=rebalance_test_table_2 [id=2ff6aeab7e1847f6884750db73f8c428]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.814388 28353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac392ff7965a405f96efc777aca4f0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.817389 28743 tablet_service.cc:1467] Processing CreateTablet for tablet ac392ff7965a405f96efc777aca4f0bb (DEFAULT_TABLE table=rebalance_test_table_2 [id=2ff6aeab7e1847f6884750db73f8c428]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.817812 28743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac392ff7965a405f96efc777aca4f0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.818418 29537 tablet_bootstrap.cc:492] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:44.820693 29537 tablet_bootstrap.cc:654] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.822768 29536 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:44.824513 29536 tablet_bootstrap.cc:654] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.827186 29536 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:44.827347 29536 ts_tablet_manager.cc:1367] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 08:12:44.828150 29536 raft_consensus.cc:348] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:44.828377 29536 raft_consensus.cc:374] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.828467 29536 raft_consensus.cc:729] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.828714 29536 consensus_queue.cc:260] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:44.832031 29536 ts_tablet_manager.cc:1398] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:44.835161 29536 tablet_bootstrap.cc:492] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:44.837023 29536 tablet_bootstrap.cc:654] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.853926 29540 tablet_bootstrap.cc:492] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:44.855856 29540 tablet_bootstrap.cc:654] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.859184 29540 tablet_bootstrap.cc:492] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:44.859424 29540 ts_tablet_manager.cc:1367] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:12:44.859625 29535 tablet_bootstrap.cc:492] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:44.860325 29540 raft_consensus.cc:348] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:44.860618 29540 raft_consensus.cc:374] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.860747 29540 raft_consensus.cc:729] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.861073 29540 consensus_queue.cc:260] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:44.861896 29540 ts_tablet_manager.cc:1398] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:44.861994 29535 tablet_bootstrap.cc:654] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.864056 29536 tablet_bootstrap.cc:492] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:44.864244 29536 ts_tablet_manager.cc:1367] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.013s
I20240503 08:12:44.864428 29540 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:44.865795 29536 raft_consensus.cc:348] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:44.866138 29536 raft_consensus.cc:374] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.866225 29536 raft_consensus.cc:729] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.866518 29536 consensus_queue.cc:260] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:44.867260 29536 ts_tablet_manager.cc:1398] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:12:44.868657 29537 tablet_bootstrap.cc:492] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:44.868835 29537 ts_tablet_manager.cc:1367] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.040s
I20240503 08:12:44.869736 29535 tablet_bootstrap.cc:492] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:44.869859 29540 tablet_bootstrap.cc:654] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.869717 29537 raft_consensus.cc:348] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:44.869944 29537 raft_consensus.cc:374] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.870048 29537 raft_consensus.cc:729] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.870157 29535 ts_tablet_manager.cc:1367] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20240503 08:12:44.870332 29537 consensus_queue.cc:260] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:44.871011 29537 ts_tablet_manager.cc:1398] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:44.871074 29535 raft_consensus.cc:348] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:44.871420 29537 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:44.871420 29535 raft_consensus.cc:374] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.871598 29535 raft_consensus.cc:729] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.871888 29535 consensus_queue.cc:260] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:44.872517 29535 ts_tablet_manager.cc:1398] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:44.873128 29537 tablet_bootstrap.cc:654] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.878868 29540 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:44.879005 29537 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:44.879041 29540 ts_tablet_manager.cc:1367] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20240503 08:12:44.879180 29537 ts_tablet_manager.cc:1367] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 08:12:44.879681 29540 raft_consensus.cc:348] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:44.879925 29540 raft_consensus.cc:374] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.880010 29540 raft_consensus.cc:729] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.880244 29540 consensus_queue.cc:260] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:44.880520 29537 raft_consensus.cc:348] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:44.880815 29537 raft_consensus.cc:374] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.881040 29537 raft_consensus.cc:729] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.881330 29540 ts_tablet_manager.cc:1398] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:44.881327 29537 consensus_queue.cc:260] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:44.881611 29540 tablet_bootstrap.cc:492] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:44.881821 29537 ts_tablet_manager.cc:1398] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:44.882061 29537 tablet_bootstrap.cc:492] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:44.883255 29537 tablet_bootstrap.cc:654] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.883849 29540 tablet_bootstrap.cc:654] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.885608 29537 tablet_bootstrap.cc:492] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:44.885747 29537 ts_tablet_manager.cc:1367] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:44.885885 29540 tablet_bootstrap.cc:492] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:44.886008 29540 ts_tablet_manager.cc:1367] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 08:12:44.886348 29537 raft_consensus.cc:348] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:44.886498 29537 raft_consensus.cc:374] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.886591 29537 raft_consensus.cc:729] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.886564 29540 raft_consensus.cc:348] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:44.886770 29540 raft_consensus.cc:374] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.886798 29537 consensus_queue.cc:260] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:44.886870 29540 raft_consensus.cc:729] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.887075 29540 consensus_queue.cc:260] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:44.887205 29537 ts_tablet_manager.cc:1398] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:44.887621 29540 ts_tablet_manager.cc:1398] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:44.897795 29399 raft_consensus.cc:482] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.897989 29399 raft_consensus.cc:504] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:44.898679 29399 leader_election.cc:288] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:44.899236 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac392ff7965a405f96efc777aca4f0bb" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" is_pre_election: true
I20240503 08:12:44.899525 28374 raft_consensus.cc:2454] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:44.899912 28439 leader_election.cc:302] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:44.900158 29399 raft_consensus.cc:2781] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.900267 29399 raft_consensus.cc:482] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.900378 29399 raft_consensus.cc:3037] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.902349 29399 raft_consensus.cc:504] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:44.902971 29399 leader_election.cc:288] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Requested vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:44.903399 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac392ff7965a405f96efc777aca4f0bb" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:12:44.903584 28374 raft_consensus.cc:3037] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.905225 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac392ff7965a405f96efc777aca4f0bb" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:44.905503 28765 raft_consensus.cc:2454] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:44.905411 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac392ff7965a405f96efc777aca4f0bb" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:44.905681 28764 raft_consensus.cc:3037] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.905956 28374 raft_consensus.cc:2454] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:44.906347 28439 leader_election.cc:302] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:44.906652 29399 raft_consensus.cc:2781] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.907060 29399 raft_consensus.cc:686] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Becoming Leader. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Running, Role: LEADER
I20240503 08:12:44.907383 29399 consensus_queue.cc:237] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:44.908071 28764 raft_consensus.cc:2454] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:44.910091 28225 catalog_manager.cc:5484] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: term changed from 0 to 1, leader changed from <none> to 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130). New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:44.940860 29367 raft_consensus.cc:482] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.941112 29367 raft_consensus.cc:504] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:44.941737 29367 leader_election.cc:288] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:12:44.942243 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13c33b8712e4c488abb99b73ff42b86" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:44.942296 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13c33b8712e4c488abb99b73ff42b86" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
I20240503 08:12:44.942478 28764 raft_consensus.cc:2454] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:44.942543 28634 raft_consensus.cc:2454] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:44.942916 28308 leader_election.cc:302] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:44.943208 29367 raft_consensus.cc:2781] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.943301 29367 raft_consensus.cc:482] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.943408 29367 raft_consensus.cc:3037] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.945303 29367 raft_consensus.cc:504] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:44.945899 29367 leader_election.cc:288] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Requested vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:12:44.946382 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13c33b8712e4c488abb99b73ff42b86" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:44.946401 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13c33b8712e4c488abb99b73ff42b86" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:44.946585 28634 raft_consensus.cc:3037] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.946581 28764 raft_consensus.cc:3037] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.948500 28764 raft_consensus.cc:2454] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:44.948652 28634 raft_consensus.cc:2454] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:44.948968 28308 leader_election.cc:302] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:44.949229 29367 raft_consensus.cc:2781] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.949416 29367 raft_consensus.cc:686] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Becoming Leader. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Running, Role: LEADER
I20240503 08:12:44.949666 29367 consensus_queue.cc:237] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:44.952257 28225 catalog_manager.cc:5484] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129). New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:44.970834 29367 raft_consensus.cc:482] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.971037 29367 raft_consensus.cc:504] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:44.971637 29367 leader_election.cc:288] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:44.972337 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9997e8da99c4f45a63bc3db521b33ba" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:44.972414 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9997e8da99c4f45a63bc3db521b33ba" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:44.972575 28504 raft_consensus.cc:2454] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:44.972656 28764 raft_consensus.cc:2454] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:44.973078 28308 leader_election.cc:302] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:44.973385 29367 raft_consensus.cc:2781] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.973513 29367 raft_consensus.cc:482] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.973623 29367 raft_consensus.cc:3037] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.975570 29367 raft_consensus.cc:504] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:44.976190 29367 leader_election.cc:288] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Requested vote from peers 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:44.976833 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9997e8da99c4f45a63bc3db521b33ba" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:12:44.976840 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9997e8da99c4f45a63bc3db521b33ba" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:44.977064 28504 raft_consensus.cc:3037] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.977073 28764 raft_consensus.cc:3037] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.979000 28764 raft_consensus.cc:2454] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:44.979000 28504 raft_consensus.cc:2454] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:44.979475 28308 leader_election.cc:302] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:44.979782 29367 raft_consensus.cc:2781] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.979965 29367 raft_consensus.cc:686] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Becoming Leader. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Running, Role: LEADER
I20240503 08:12:44.980265 29367 consensus_queue.cc:237] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:44.982849 28225 catalog_manager.cc:5484] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129). New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:45.339852 29399 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.344645 29555 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:45.378662 29553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:45.379158 29553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:45.386832 29553 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:45.397742 29367 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.402645 29570 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 08:12:45.496601 29570 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.509397 29367 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 222.658 ms
time per row: 0.111329 ms
I20240503 08:12:45.759157 28231 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60904:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:45.769730 28614 tablet_service.cc:1467] Processing CreateTablet for tablet d60a162155504975a39c20eff88341e1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eeb192513dc04f0eb0552ff611d314c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.769951 28353 tablet_service.cc:1467] Processing CreateTablet for tablet d60a162155504975a39c20eff88341e1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eeb192513dc04f0eb0552ff611d314c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.770221 28614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60a162155504975a39c20eff88341e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.770426 28353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60a162155504975a39c20eff88341e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.771754 28483 tablet_service.cc:1467] Processing CreateTablet for tablet d60a162155504975a39c20eff88341e1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eeb192513dc04f0eb0552ff611d314c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.772168 28483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60a162155504975a39c20eff88341e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.772264 28745 tablet_service.cc:1467] Processing CreateTablet for tablet a8b4e398d34443b4a440f9ba48555d84 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eeb192513dc04f0eb0552ff611d314c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.772661 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b4e398d34443b4a440f9ba48555d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.773667 28354 tablet_service.cc:1467] Processing CreateTablet for tablet 28e42c3244f949c7b7343bf587894ac5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eeb192513dc04f0eb0552ff611d314c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.773754 28613 tablet_service.cc:1467] Processing CreateTablet for tablet a8b4e398d34443b4a440f9ba48555d84 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eeb192513dc04f0eb0552ff611d314c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.774076 28354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e42c3244f949c7b7343bf587894ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.774173 28613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b4e398d34443b4a440f9ba48555d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.776332 28484 tablet_service.cc:1467] Processing CreateTablet for tablet a8b4e398d34443b4a440f9ba48555d84 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eeb192513dc04f0eb0552ff611d314c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.776705 28484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b4e398d34443b4a440f9ba48555d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.781301 28743 tablet_service.cc:1467] Processing CreateTablet for tablet 28e42c3244f949c7b7343bf587894ac5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eeb192513dc04f0eb0552ff611d314c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.781718 28743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e42c3244f949c7b7343bf587894ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.785414 28615 tablet_service.cc:1467] Processing CreateTablet for tablet 28e42c3244f949c7b7343bf587894ac5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eeb192513dc04f0eb0552ff611d314c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.785967 28615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e42c3244f949c7b7343bf587894ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.789798 29619 tablet_bootstrap.cc:492] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:45.792060 29619 tablet_bootstrap.cc:654] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.795128 29618 tablet_bootstrap.cc:492] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:45.800256 29618 tablet_bootstrap.cc:654] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.800352 29619 tablet_bootstrap.cc:492] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:45.800537 29619 ts_tablet_manager.cc:1367] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240503 08:12:45.801393 29619 raft_consensus.cc:348] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:45.801766 29619 raft_consensus.cc:374] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.801898 29619 raft_consensus.cc:729] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.802213 29619 consensus_queue.cc:260] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:45.805584 29619 ts_tablet_manager.cc:1398] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:45.808069 29619 tablet_bootstrap.cc:492] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:45.809746 29619 tablet_bootstrap.cc:654] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.814267 29623 tablet_bootstrap.cc:492] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:45.817912 29619 tablet_bootstrap.cc:492] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:45.818152 29619 ts_tablet_manager.cc:1367] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240503 08:12:45.818946 29619 raft_consensus.cc:348] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.819236 29619 raft_consensus.cc:374] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.819344 29619 raft_consensus.cc:729] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.819651 29619 consensus_queue.cc:260] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.819837 29623 tablet_bootstrap.cc:654] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.823455 29619 ts_tablet_manager.cc:1398] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:12:45.830478 29623 tablet_bootstrap.cc:492] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:45.830706 29623 ts_tablet_manager.cc:1367] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20240503 08:12:45.831847 29623 raft_consensus.cc:348] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.832144 29623 raft_consensus.cc:374] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.832258 29623 raft_consensus.cc:729] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.834874 29617 tablet_bootstrap.cc:492] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:45.835069 29618 tablet_bootstrap.cc:492] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:45.835238 29618 ts_tablet_manager.cc:1367] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.013s
I20240503 08:12:45.836027 29618 raft_consensus.cc:348] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.836194 29618 raft_consensus.cc:374] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.836290 29618 raft_consensus.cc:729] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.836544 29618 consensus_queue.cc:260] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.836681 29617 tablet_bootstrap.cc:654] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.837098 29618 ts_tablet_manager.cc:1398] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:45.837386 29618 tablet_bootstrap.cc:492] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:45.838933 29618 tablet_bootstrap.cc:654] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.834187 29623 consensus_queue.cc:260] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.841032 29623 ts_tablet_manager.cc:1398] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20240503 08:12:45.841339 29623 tablet_bootstrap.cc:492] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:45.843864 29623 tablet_bootstrap.cc:654] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.846010 29623 tablet_bootstrap.cc:492] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:45.846177 29623 ts_tablet_manager.cc:1367] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:12:45.846937 29623 raft_consensus.cc:348] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.847137 29623 raft_consensus.cc:374] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.847215 29623 raft_consensus.cc:729] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.847458 29623 consensus_queue.cc:260] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.849247 29623 ts_tablet_manager.cc:1398] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:45.850515 29618 tablet_bootstrap.cc:492] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:45.850687 29618 ts_tablet_manager.cc:1367] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.004s
I20240503 08:12:45.851475 29618 raft_consensus.cc:348] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.851698 29618 raft_consensus.cc:374] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.851778 29618 raft_consensus.cc:729] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.852015 29618 consensus_queue.cc:260] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.852551 29618 ts_tablet_manager.cc:1398] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:45.852850 29618 tablet_bootstrap.cc:492] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:45.854409 29618 tablet_bootstrap.cc:654] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.855949 29617 tablet_bootstrap.cc:492] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:45.856110 29617 ts_tablet_manager.cc:1367] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.007s
I20240503 08:12:45.856913 29617 raft_consensus.cc:348] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.857132 29617 raft_consensus.cc:374] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.857213 29617 raft_consensus.cc:729] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.857425 29617 consensus_queue.cc:260] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.857945 29617 ts_tablet_manager.cc:1398] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:45.857961 29618 tablet_bootstrap.cc:492] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:45.858106 29618 ts_tablet_manager.cc:1367] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 08:12:45.858541 29617 tablet_bootstrap.cc:492] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:45.858834 29618 raft_consensus.cc:348] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:45.859026 29618 raft_consensus.cc:374] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.859113 29618 raft_consensus.cc:729] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.859386 29618 consensus_queue.cc:260] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:45.860054 29618 ts_tablet_manager.cc:1398] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:45.860080 29617 tablet_bootstrap.cc:654] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.863162 29617 tablet_bootstrap.cc:492] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:45.863312 29617 ts_tablet_manager.cc:1367] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 08:12:45.864043 29617 raft_consensus.cc:348] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:45.864305 29617 raft_consensus.cc:374] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.864461 29617 raft_consensus.cc:729] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.864729 29617 consensus_queue.cc:260] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:45.865159 29617 ts_tablet_manager.cc:1398] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:45.869726 29567 raft_consensus.cc:482] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:45.869901 29567 raft_consensus.cc:504] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:45.870608 29567 leader_election.cc:288] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:12:45.871932 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e42c3244f949c7b7343bf587894ac5" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" is_pre_election: true
I20240503 08:12:45.872233 28374 raft_consensus.cc:2454] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 0.
I20240503 08:12:45.872783 28569 leader_election.cc:302] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 4e4a1242f3714d03a6c9ee51dd63929c; no voters:
I20240503 08:12:45.873111 29567 raft_consensus.cc:2781] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:45.873270 29567 raft_consensus.cc:482] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:45.873385 29567 raft_consensus.cc:3037] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.875985 29567 raft_consensus.cc:504] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:45.876686 29567 leader_election.cc:288] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 election: Requested vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:12:45.877344 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e42c3244f949c7b7343bf587894ac5" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:12:45.877554 28374 raft_consensus.cc:3037] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.878542 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e42c3244f949c7b7343bf587894ac5" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:45.878636 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e42c3244f949c7b7343bf587894ac5" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:45.878798 28764 raft_consensus.cc:2454] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 0.
I20240503 08:12:45.879885 28374 raft_consensus.cc:2454] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 1.
I20240503 08:12:45.880460 28569 leader_election.cc:302] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 4e4a1242f3714d03a6c9ee51dd63929c; no voters: 93f7e733cec9415eb422b0a7ba8621cb
I20240503 08:12:45.880789 29567 raft_consensus.cc:2781] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:45.881273 29567 raft_consensus.cc:686] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Becoming Leader. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Running, Role: LEADER
I20240503 08:12:45.881716 29567 consensus_queue.cc:237] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:45.885560 28224 catalog_manager.cc:5484] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4574396f35514c9caccb346f1a7af84d (127.20.107.131). New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:45.967337 29567 raft_consensus.cc:482] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:45.967600 29567 raft_consensus.cc:504] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.968168 29567 leader_election.cc:288] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:45.968998 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60a162155504975a39c20eff88341e1" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" is_pre_election: true
I20240503 08:12:45.969225 28374 raft_consensus.cc:2454] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 0.
I20240503 08:12:45.969147 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60a162155504975a39c20eff88341e1" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:45.969462 28504 raft_consensus.cc:2454] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 0.
I20240503 08:12:45.969791 28569 leader_election.cc:302] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 4e4a1242f3714d03a6c9ee51dd63929c; no voters:
I20240503 08:12:45.970084 29567 raft_consensus.cc:2781] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:45.970192 29567 raft_consensus.cc:482] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:45.970301 29567 raft_consensus.cc:3037] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.971570 29549 raft_consensus.cc:482] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:45.971738 29549 raft_consensus.cc:504] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.972292 29567 raft_consensus.cc:504] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.972431 29549 leader_election.cc:288] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:45.972949 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b4e398d34443b4a440f9ba48555d84" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
I20240503 08:12:45.972988 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b4e398d34443b4a440f9ba48555d84" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:45.973199 28634 raft_consensus.cc:2454] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:45.973269 28764 raft_consensus.cc:2454] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:45.973726 28438 leader_election.cc:302] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:45.973825 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60a162155504975a39c20eff88341e1" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:12:45.973894 29567 leader_election.cc:288] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 election: Requested vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:45.974047 28374 raft_consensus.cc:3037] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.974277 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60a162155504975a39c20eff88341e1" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:12:45.974454 28504 raft_consensus.cc:3037] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.976125 28374 raft_consensus.cc:2454] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 1.
I20240503 08:12:45.976217 28504 raft_consensus.cc:2454] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 1.
I20240503 08:12:45.976626 29549 raft_consensus.cc:2781] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:45.976593 28569 leader_election.cc:302] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 4e4a1242f3714d03a6c9ee51dd63929c; no voters:
I20240503 08:12:45.976724 29549 raft_consensus.cc:482] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:45.976817 29549 raft_consensus.cc:3037] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.976974 29567 raft_consensus.cc:2781] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:45.977169 29567 raft_consensus.cc:686] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Becoming Leader. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Running, Role: LEADER
I20240503 08:12:45.977435 29567 consensus_queue.cc:237] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.979193 29549 raft_consensus.cc:504] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.979827 29549 leader_election.cc:288] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Requested vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:45.980216 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b4e398d34443b4a440f9ba48555d84" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:45.980275 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b4e398d34443b4a440f9ba48555d84" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:45.980459 28634 raft_consensus.cc:3037] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.980499 28764 raft_consensus.cc:3037] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.980311 28224 catalog_manager.cc:5484] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4574396f35514c9caccb346f1a7af84d (127.20.107.131). New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:45.983335 28764 raft_consensus.cc:2454] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:45.983335 28634 raft_consensus.cc:2454] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:45.983850 28438 leader_election.cc:302] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 716ffbb2a00b4f43b03e5cedcc9d302a, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:45.984167 29549 raft_consensus.cc:2781] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:45.984321 29549 raft_consensus.cc:686] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Becoming Leader. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Running, Role: LEADER
I20240503 08:12:45.984583 29549 consensus_queue.cc:237] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:45.987262 28224 catalog_manager.cc:5484] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: term changed from 0 to 1, leader changed from <none> to 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130). New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:46.320195 29567 consensus_queue.cc:1035] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.325102 28764 raft_consensus.cc:3037] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.329296 29639 consensus_queue.cc:1035] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:46.406352 29636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:46.406694 29636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:46.414450 29636 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:46.445390 29555 consensus_queue.cc:1035] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240503 08:12:46.450982 29549 consensus_queue.cc:1035] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.477026 29639 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.482136 29567 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 314.907 ms
time per row: 0.157453 ms
I20240503 08:12:46.890556 28224 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60904:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:46.899983 28615 tablet_service.cc:1467] Processing CreateTablet for tablet 173376257c4946b5a801a27e4dffe90d (DEFAULT_TABLE table=rebalance_test_table_4 [id=3a90dbe17357489db92bd286b71aa565]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.899938 28353 tablet_service.cc:1467] Processing CreateTablet for tablet 173376257c4946b5a801a27e4dffe90d (DEFAULT_TABLE table=rebalance_test_table_4 [id=3a90dbe17357489db92bd286b71aa565]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.900420 28353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 173376257c4946b5a801a27e4dffe90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.900419 28615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 173376257c4946b5a801a27e4dffe90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.900676 28483 tablet_service.cc:1467] Processing CreateTablet for tablet 173376257c4946b5a801a27e4dffe90d (DEFAULT_TABLE table=rebalance_test_table_4 [id=3a90dbe17357489db92bd286b71aa565]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.901077 28483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 173376257c4946b5a801a27e4dffe90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.905859 28743 tablet_service.cc:1467] Processing CreateTablet for tablet 09156806b0744752969b6a574e24b50a (DEFAULT_TABLE table=rebalance_test_table_4 [id=3a90dbe17357489db92bd286b71aa565]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.905918 28745 tablet_service.cc:1467] Processing CreateTablet for tablet b0d07a6278d545e69bf7d2012109d1f4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3a90dbe17357489db92bd286b71aa565]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.906334 28743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09156806b0744752969b6a574e24b50a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.906495 28484 tablet_service.cc:1467] Processing CreateTablet for tablet b0d07a6278d545e69bf7d2012109d1f4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3a90dbe17357489db92bd286b71aa565]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.906472 28354 tablet_service.cc:1467] Processing CreateTablet for tablet b0d07a6278d545e69bf7d2012109d1f4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3a90dbe17357489db92bd286b71aa565]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.906855 28484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d07a6278d545e69bf7d2012109d1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.906957 28354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d07a6278d545e69bf7d2012109d1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.910385 28352 tablet_service.cc:1467] Processing CreateTablet for tablet 09156806b0744752969b6a574e24b50a (DEFAULT_TABLE table=rebalance_test_table_4 [id=3a90dbe17357489db92bd286b71aa565]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.910931 28352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09156806b0744752969b6a574e24b50a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.913779 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d07a6278d545e69bf7d2012109d1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.921087 29704 tablet_bootstrap.cc:492] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:46.923039 29704 tablet_bootstrap.cc:654] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.923957 28613 tablet_service.cc:1467] Processing CreateTablet for tablet 09156806b0744752969b6a574e24b50a (DEFAULT_TABLE table=rebalance_test_table_4 [id=3a90dbe17357489db92bd286b71aa565]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.924418 28613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09156806b0744752969b6a574e24b50a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.927505 29703 tablet_bootstrap.cc:492] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:46.927593 29702 tablet_bootstrap.cc:492] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:46.928857 29704 tablet_bootstrap.cc:492] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:46.929087 29704 ts_tablet_manager.cc:1367] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 08:12:46.929605 29703 tablet_bootstrap.cc:654] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.929626 29702 tablet_bootstrap.cc:654] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.930039 29704 raft_consensus.cc:348] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:46.930292 29704 raft_consensus.cc:374] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.930428 29704 raft_consensus.cc:729] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.930809 29704 consensus_queue.cc:260] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:46.931696 29704 ts_tablet_manager.cc:1398] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:46.935916 29704 tablet_bootstrap.cc:492] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:46.936609 29708 tablet_bootstrap.cc:492] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:46.940372 29708 tablet_bootstrap.cc:654] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.943259 29702 tablet_bootstrap.cc:492] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:46.943531 29702 ts_tablet_manager.cc:1367] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20240503 08:12:46.943615 29704 tablet_bootstrap.cc:654] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.944687 29702 raft_consensus.cc:348] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:46.945092 29702 raft_consensus.cc:374] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.945264 29702 raft_consensus.cc:729] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.945628 29702 consensus_queue.cc:260] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:46.946564 29708 tablet_bootstrap.cc:492] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:46.946766 29708 ts_tablet_manager.cc:1367] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240503 08:12:46.947813 29703 tablet_bootstrap.cc:492] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:46.947700 29708 raft_consensus.cc:348] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:46.947868 29704 tablet_bootstrap.cc:492] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:46.947902 29708 raft_consensus.cc:374] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.948056 29703 ts_tablet_manager.cc:1367] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.014s
I20240503 08:12:46.948066 29704 ts_tablet_manager.cc:1367] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.006s
I20240503 08:12:46.948068 29708 raft_consensus.cc:729] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.948469 29708 consensus_queue.cc:260] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:46.948814 29704 raft_consensus.cc:348] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:46.949123 29704 raft_consensus.cc:374] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.949251 29708 ts_tablet_manager.cc:1398] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:46.949239 29702 ts_tablet_manager.cc:1398] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:46.949263 29704 raft_consensus.cc:729] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.949551 29708 tablet_bootstrap.cc:492] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:46.949668 29704 consensus_queue.cc:260] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:46.950311 29704 ts_tablet_manager.cc:1398] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:46.950668 29703 raft_consensus.cc:348] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:46.950960 29703 raft_consensus.cc:374] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.951109 29703 raft_consensus.cc:729] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.951164 29708 tablet_bootstrap.cc:654] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.951458 29703 consensus_queue.cc:260] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:46.952024 29703 ts_tablet_manager.cc:1398] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:12:46.952453 29703 tablet_bootstrap.cc:492] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:46.953442 29702 tablet_bootstrap.cc:492] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:46.954321 29703 tablet_bootstrap.cc:654] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.957304 29702 tablet_bootstrap.cc:654] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.960131 29708 tablet_bootstrap.cc:492] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:46.960368 29708 ts_tablet_manager.cc:1367] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240503 08:12:46.961419 29703 tablet_bootstrap.cc:492] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:46.961334 29708 raft_consensus.cc:348] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:46.961555 29708 raft_consensus.cc:374] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.961656 29703 ts_tablet_manager.cc:1367] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 08:12:46.961650 29708 raft_consensus.cc:729] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.961920 29702 tablet_bootstrap.cc:492] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:46.961896 29708 consensus_queue.cc:260] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:46.962077 29702 ts_tablet_manager.cc:1367] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20240503 08:12:46.962461 29708 ts_tablet_manager.cc:1398] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:46.962837 29702 raft_consensus.cc:348] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:46.962935 29703 raft_consensus.cc:348] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:46.963059 29702 raft_consensus.cc:374] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.963132 29702 raft_consensus.cc:729] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.963160 29703 raft_consensus.cc:374] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.963228 29703 raft_consensus.cc:729] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.963402 29702 consensus_queue.cc:260] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:46.963474 29703 consensus_queue.cc:260] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:46.964004 29703 ts_tablet_manager.cc:1398] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:46.964653 29702 ts_tablet_manager.cc:1398] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:46.965018 29702 tablet_bootstrap.cc:492] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:46.966895 29702 tablet_bootstrap.cc:654] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.971112 29702 tablet_bootstrap.cc:492] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:46.971256 29702 ts_tablet_manager.cc:1367] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240503 08:12:46.971462 29549 raft_consensus.cc:482] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:46.971603 29549 raft_consensus.cc:504] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:46.971928 29702 raft_consensus.cc:348] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:46.972100 29702 raft_consensus.cc:374] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.972189 29702 raft_consensus.cc:729] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.972407 29549 leader_election.cc:288] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:12:46.972411 29702 consensus_queue.cc:260] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:46.972844 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d07a6278d545e69bf7d2012109d1f4" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:46.973059 29702 ts_tablet_manager.cc:1398] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:46.973096 28765 raft_consensus.cc:2454] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:46.973059 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d07a6278d545e69bf7d2012109d1f4" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" is_pre_election: true
I20240503 08:12:46.973374 28374 raft_consensus.cc:2454] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:46.973536 28439 leader_election.cc:302] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716ffbb2a00b4f43b03e5cedcc9d302a, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:46.973855 29549 raft_consensus.cc:2781] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:46.973985 29549 raft_consensus.cc:482] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:46.974082 29549 raft_consensus.cc:3037] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.976542 29549 raft_consensus.cc:504] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:46.977110 29549 leader_election.cc:288] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Requested vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:12:46.977625 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d07a6278d545e69bf7d2012109d1f4" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:46.977675 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d07a6278d545e69bf7d2012109d1f4" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:12:46.977867 28765 raft_consensus.cc:3037] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.977887 28374 raft_consensus.cc:3037] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.980720 28374 raft_consensus.cc:2454] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:46.980746 28765 raft_consensus.cc:2454] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:46.981204 28439 leader_election.cc:302] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716ffbb2a00b4f43b03e5cedcc9d302a, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:46.981487 29549 raft_consensus.cc:2781] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:46.981678 29549 raft_consensus.cc:686] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Becoming Leader. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Running, Role: LEADER
I20240503 08:12:46.981957 29549 consensus_queue.cc:237] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:46.984697 28231 catalog_manager.cc:5484] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: term changed from 0 to 1, leader changed from <none> to 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130). New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:47.053723 29608 raft_consensus.cc:482] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:47.053921 29608 raft_consensus.cc:504] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:47.054600 29608 leader_election.cc:288] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397), 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:12:47.055241 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "173376257c4946b5a801a27e4dffe90d" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:47.055234 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "173376257c4946b5a801a27e4dffe90d" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
I20240503 08:12:47.055481 28504 raft_consensus.cc:2454] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:47.055482 28634 raft_consensus.cc:2454] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:47.056134 28307 leader_election.cc:302] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:47.056444 29608 raft_consensus.cc:2781] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:47.056571 29608 raft_consensus.cc:482] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:47.056643 29608 raft_consensus.cc:3037] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.058554 29608 raft_consensus.cc:504] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:47.059134 29608 leader_election.cc:288] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Requested vote from peers 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397), 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:12:47.059705 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "173376257c4946b5a801a27e4dffe90d" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:12:47.059720 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "173376257c4946b5a801a27e4dffe90d" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:47.059886 28504 raft_consensus.cc:3037] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.059911 28634 raft_consensus.cc:3037] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.061820 28504 raft_consensus.cc:2454] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:47.061913 28634 raft_consensus.cc:2454] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:47.062311 28308 leader_election.cc:302] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:47.062654 29608 raft_consensus.cc:2781] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:47.063023 29608 raft_consensus.cc:686] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Becoming Leader. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Running, Role: LEADER
I20240503 08:12:47.063349 29608 consensus_queue.cc:237] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:47.066373 28231 catalog_manager.cc:5484] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129). New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:47.277127 29686 raft_consensus.cc:482] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:47.277351 29686 raft_consensus.cc:504] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:47.278450 29686 leader_election.cc:288] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:47.279098 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09156806b0744752969b6a574e24b50a" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" is_pre_election: true
I20240503 08:12:47.279162 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09156806b0744752969b6a574e24b50a" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:47.279323 28374 raft_consensus.cc:2454] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 0.
I20240503 08:12:47.279422 28765 raft_consensus.cc:2454] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 0.
I20240503 08:12:47.279758 28569 leader_election.cc:302] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 4e4a1242f3714d03a6c9ee51dd63929c; no voters:
I20240503 08:12:47.280071 29686 raft_consensus.cc:2781] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:47.280179 29686 raft_consensus.cc:482] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:47.280283 29686 raft_consensus.cc:3037] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.282508 29686 raft_consensus.cc:504] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:47.283147 29686 leader_election.cc:288] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 election: Requested vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:47.283723 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09156806b0744752969b6a574e24b50a" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:47.283941 28765 raft_consensus.cc:3037] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.286240 28765 raft_consensus.cc:2454] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 1.
I20240503 08:12:47.286711 28569 leader_election.cc:302] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:47.287019 29686 raft_consensus.cc:2781] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:47.287160 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09156806b0744752969b6a574e24b50a" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:12:47.287506 28374 raft_consensus.cc:3037] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.287549 29686 raft_consensus.cc:686] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Becoming Leader. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Running, Role: LEADER
I20240503 08:12:47.287840 29686 consensus_queue.cc:237] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:47.290119 28374 raft_consensus.cc:2454] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 1.
I20240503 08:12:47.290947 28231 catalog_manager.cc:5484] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4574396f35514c9caccb346f1a7af84d (127.20.107.131). New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:47.409811 29549 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.417078 29549 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.526988 29608 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.532143 29608 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:47.737067 29720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:47.737475 29720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:47.745388 29720 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 08:12:47.819237 29686 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.836371 29754 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 305.798 ms
time per row: 0.152899 ms
I20240503 08:12:48.215145 28228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60904:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:48.223966 28615 tablet_service.cc:1467] Processing CreateTablet for tablet 4babf0ea9b954a23b1b1c4e403dce0bd (DEFAULT_TABLE table=rebalance_test_table_5 [id=311fe8a0590b44a8a07f0203d8f65e32]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.224100 28483 tablet_service.cc:1467] Processing CreateTablet for tablet 4babf0ea9b954a23b1b1c4e403dce0bd (DEFAULT_TABLE table=rebalance_test_table_5 [id=311fe8a0590b44a8a07f0203d8f65e32]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.224494 28615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4babf0ea9b954a23b1b1c4e403dce0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.224537 28483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4babf0ea9b954a23b1b1c4e403dce0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.225090 28745 tablet_service.cc:1467] Processing CreateTablet for tablet 320d1e880958427a88e93321171cb36e (DEFAULT_TABLE table=rebalance_test_table_5 [id=311fe8a0590b44a8a07f0203d8f65e32]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.225492 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320d1e880958427a88e93321171cb36e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.225857 28743 tablet_service.cc:1467] Processing CreateTablet for tablet 4babf0ea9b954a23b1b1c4e403dce0bd (DEFAULT_TABLE table=rebalance_test_table_5 [id=311fe8a0590b44a8a07f0203d8f65e32]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.225912 28484 tablet_service.cc:1467] Processing CreateTablet for tablet 320d1e880958427a88e93321171cb36e (DEFAULT_TABLE table=rebalance_test_table_5 [id=311fe8a0590b44a8a07f0203d8f65e32]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.226226 28743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4babf0ea9b954a23b1b1c4e403dce0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.226560 28485 tablet_service.cc:1467] Processing CreateTablet for tablet d44335d8be7245b1bb52b2eab9e42c18 (DEFAULT_TABLE table=rebalance_test_table_5 [id=311fe8a0590b44a8a07f0203d8f65e32]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.229089 28484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320d1e880958427a88e93321171cb36e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.232455 28353 tablet_service.cc:1467] Processing CreateTablet for tablet d44335d8be7245b1bb52b2eab9e42c18 (DEFAULT_TABLE table=rebalance_test_table_5 [id=311fe8a0590b44a8a07f0203d8f65e32]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.232916 28353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44335d8be7245b1bb52b2eab9e42c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.232997 28485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44335d8be7245b1bb52b2eab9e42c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.234309 28614 tablet_service.cc:1467] Processing CreateTablet for tablet d44335d8be7245b1bb52b2eab9e42c18 (DEFAULT_TABLE table=rebalance_test_table_5 [id=311fe8a0590b44a8a07f0203d8f65e32]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.234547 29790 tablet_bootstrap.cc:492] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:48.234740 28614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44335d8be7245b1bb52b2eab9e42c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.236294 29790 tablet_bootstrap.cc:654] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.237294 28613 tablet_service.cc:1467] Processing CreateTablet for tablet 320d1e880958427a88e93321171cb36e (DEFAULT_TABLE table=rebalance_test_table_5 [id=311fe8a0590b44a8a07f0203d8f65e32]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.237686 28613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320d1e880958427a88e93321171cb36e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.249491 29792 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:48.249573 29793 tablet_bootstrap.cc:492] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:48.251993 29792 tablet_bootstrap.cc:654] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.253602 29793 tablet_bootstrap.cc:654] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.260324 29792 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:48.260586 29792 ts_tablet_manager.cc:1367] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240503 08:12:48.261757 29792 raft_consensus.cc:348] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:48.262099 29789 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:48.262157 29792 raft_consensus.cc:374] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.262320 29792 raft_consensus.cc:729] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.262658 29792 consensus_queue.cc:260] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:48.264089 29789 tablet_bootstrap.cc:654] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.264289 29792 ts_tablet_manager.cc:1398] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:48.265455 29792 tablet_bootstrap.cc:492] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:48.265914 29793 tablet_bootstrap.cc:492] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:48.266178 29793 ts_tablet_manager.cc:1367] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20240503 08:12:48.267506 29793 raft_consensus.cc:348] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.267994 29792 tablet_bootstrap.cc:654] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.268011 29793 raft_consensus.cc:374] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.268203 29793 raft_consensus.cc:729] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.268572 29793 consensus_queue.cc:260] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.269397 29793 ts_tablet_manager.cc:1398] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:48.271029 29790 tablet_bootstrap.cc:492] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:48.271281 29790 ts_tablet_manager.cc:1367] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.010s
I20240503 08:12:48.272251 29790 raft_consensus.cc:348] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.272572 29790 raft_consensus.cc:374] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.272734 29790 raft_consensus.cc:729] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.273072 29790 consensus_queue.cc:260] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.273787 29790 ts_tablet_manager.cc:1398] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:48.274116 29790 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:48.274279 29789 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:48.275035 29789 ts_tablet_manager.cc:1367] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.005s
I20240503 08:12:48.275848 29790 tablet_bootstrap.cc:654] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.276194 29789 raft_consensus.cc:348] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:48.276477 29789 raft_consensus.cc:374] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.276623 29789 raft_consensus.cc:729] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.276978 29789 consensus_queue.cc:260] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:48.277594 29789 ts_tablet_manager.cc:1398] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:48.278334 29792 tablet_bootstrap.cc:492] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:48.278540 29792 ts_tablet_manager.cc:1367] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.001s
I20240503 08:12:48.279356 29792 raft_consensus.cc:348] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.279613 29792 raft_consensus.cc:374] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.279712 29792 raft_consensus.cc:729] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.280838 29792 consensus_queue.cc:260] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.282069 29792 ts_tablet_manager.cc:1398] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:48.282423 29792 tablet_bootstrap.cc:492] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:48.282689 29789 tablet_bootstrap.cc:492] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:48.284276 29789 tablet_bootstrap.cc:654] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.286136 29792 tablet_bootstrap.cc:654] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.290799 29790 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:48.291002 29790 ts_tablet_manager.cc:1367] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20240503 08:12:48.291896 29790 raft_consensus.cc:348] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:48.292132 29790 raft_consensus.cc:374] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.292212 29790 raft_consensus.cc:729] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.293742 29790 consensus_queue.cc:260] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:48.294342 29790 ts_tablet_manager.cc:1398] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:48.295680 29789 tablet_bootstrap.cc:492] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:48.295873 29789 ts_tablet_manager.cc:1367] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20240503 08:12:48.296666 29789 raft_consensus.cc:348] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.296980 29789 raft_consensus.cc:374] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.297111 29789 raft_consensus.cc:729] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.297441 29789 consensus_queue.cc:260] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.298143 29789 ts_tablet_manager.cc:1398] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:48.298492 29789 tablet_bootstrap.cc:492] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:48.300200 29789 tablet_bootstrap.cc:654] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.300345 29792 tablet_bootstrap.cc:492] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:48.300644 29792 ts_tablet_manager.cc:1367] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20240503 08:12:48.302979 29792 raft_consensus.cc:348] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.305255 29792 raft_consensus.cc:374] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.305393 29792 raft_consensus.cc:729] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.305698 29792 consensus_queue.cc:260] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.306438 29792 ts_tablet_manager.cc:1398] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:48.307968 29789 tablet_bootstrap.cc:492] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:48.308183 29789 ts_tablet_manager.cc:1367] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240503 08:12:48.309284 29789 raft_consensus.cc:348] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.309495 29789 raft_consensus.cc:374] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.309631 29789 raft_consensus.cc:729] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.309873 29789 consensus_queue.cc:260] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.310443 29789 ts_tablet_manager.cc:1398] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:48.322222 29754 raft_consensus.cc:482] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:48.322446 29754 raft_consensus.cc:504] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.323097 29754 leader_election.cc:288] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397), 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:12:48.323601 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44335d8be7245b1bb52b2eab9e42c18" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:48.323714 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44335d8be7245b1bb52b2eab9e42c18" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" is_pre_election: true
I20240503 08:12:48.323846 28504 raft_consensus.cc:2454] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 0.
I20240503 08:12:48.323987 28374 raft_consensus.cc:2454] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 0.
I20240503 08:12:48.324288 28569 leader_election.cc:302] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:48.324620 29754 raft_consensus.cc:2781] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:48.324705 29754 raft_consensus.cc:482] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:48.324810 29754 raft_consensus.cc:3037] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.326866 29754 raft_consensus.cc:504] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.327389 29754 leader_election.cc:288] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 election: Requested vote from peers 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397), 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:12:48.327822 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44335d8be7245b1bb52b2eab9e42c18" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:12:48.327967 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44335d8be7245b1bb52b2eab9e42c18" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:12:48.328037 28504 raft_consensus.cc:3037] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.328177 28374 raft_consensus.cc:3037] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.329978 28504 raft_consensus.cc:2454] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 1.
I20240503 08:12:48.329982 28374 raft_consensus.cc:2454] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 1.
I20240503 08:12:48.330400 28569 leader_election.cc:302] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:48.330672 29754 raft_consensus.cc:2781] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:48.330874 29754 raft_consensus.cc:686] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Becoming Leader. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Running, Role: LEADER
I20240503 08:12:48.331115 29754 consensus_queue.cc:237] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.334137 28228 catalog_manager.cc:5484] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4574396f35514c9caccb346f1a7af84d (127.20.107.131). New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:48.390821 29483 raft_consensus.cc:482] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:48.391055 29483 raft_consensus.cc:504] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:48.391686 29483 leader_election.cc:288] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:48.392225 28635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4babf0ea9b954a23b1b1c4e403dce0bd" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
I20240503 08:12:48.392292 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4babf0ea9b954a23b1b1c4e403dce0bd" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:48.392494 28635 raft_consensus.cc:2454] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 0.
I20240503 08:12:48.392531 28504 raft_consensus.cc:2454] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 0.
I20240503 08:12:48.392976 28699 leader_election.cc:302] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716ffbb2a00b4f43b03e5cedcc9d302a, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:48.393296 29483 raft_consensus.cc:2781] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:48.393431 29483 raft_consensus.cc:482] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:48.393680 29483 raft_consensus.cc:3037] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.395579 29483 raft_consensus.cc:504] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:48.396184 29483 leader_election.cc:288] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 election: Requested vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:48.396709 28635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4babf0ea9b954a23b1b1c4e403dce0bd" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:48.396759 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4babf0ea9b954a23b1b1c4e403dce0bd" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:12:48.396991 28504 raft_consensus.cc:3037] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.397001 28635 raft_consensus.cc:3037] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.399081 28504 raft_consensus.cc:2454] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 1.
I20240503 08:12:48.399502 28699 leader_election.cc:302] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716ffbb2a00b4f43b03e5cedcc9d302a, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:48.399832 29483 raft_consensus.cc:2781] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:48.400290 29483 raft_consensus.cc:686] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Becoming Leader. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Running, Role: LEADER
I20240503 08:12:48.400544 29483 consensus_queue.cc:237] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:48.403712 28228 catalog_manager.cc:5484] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132). New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:48.410781 29549 raft_consensus.cc:482] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:48.410995 29549 raft_consensus.cc:504] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.411710 29549 leader_election.cc:288] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:12:48.412137 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320d1e880958427a88e93321171cb36e" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:48.412206 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320d1e880958427a88e93321171cb36e" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
I20240503 08:12:48.412367 28764 raft_consensus.cc:2454] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:48.412432 28634 raft_consensus.cc:2454] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:48.412775 28439 leader_election.cc:302] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716ffbb2a00b4f43b03e5cedcc9d302a, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:48.413098 29549 raft_consensus.cc:2781] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:48.413205 29549 raft_consensus.cc:482] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:48.413290 29549 raft_consensus.cc:3037] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.415084 29549 raft_consensus.cc:504] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.415714 29549 leader_election.cc:288] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Requested vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:12:48.416075 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320d1e880958427a88e93321171cb36e" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:48.416216 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320d1e880958427a88e93321171cb36e" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:48.416314 28764 raft_consensus.cc:3037] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.416455 28634 raft_consensus.cc:3037] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.418605 28764 raft_consensus.cc:2454] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:48.418980 28439 leader_election.cc:302] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716ffbb2a00b4f43b03e5cedcc9d302a, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:48.419276 29549 raft_consensus.cc:2781] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:48.419422 29549 raft_consensus.cc:686] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Becoming Leader. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Running, Role: LEADER
I20240503 08:12:48.419705 29549 consensus_queue.cc:237] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:48.423071 28635 raft_consensus.cc:2454] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 1.
I20240503 08:12:48.423391 28634 raft_consensus.cc:2454] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:48.424593 28228 catalog_manager.cc:5484] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: term changed from 0 to 1, leader changed from <none> to 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130). New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:48.811183 29549 consensus_queue.cc:1035] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.814174 29483 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:48.822266 29806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:48.822866 29806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:12:48.837821 29555 consensus_queue.cc:1035] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:48.839242 29806 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:48.843281 29754 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.870879 29803 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.884193 29754 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
W20240503 08:12:49.139067 28569 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.130:40397, user_credentials={real_user=slave}} blocked reactor thread for 46204us
INSERT report
rows total: 2000
time total: 272.792 ms
time per row: 0.136396 ms
I20240503 08:12:49.328163 28230 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60904:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:49.338230 28743 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3b8b80ebd44f27899f97211266c7b2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=87e8ca2152ac403cb71e5eb67c0b2e0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.338735 28743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3b8b80ebd44f27899f97211266c7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.338781 28483 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3b8b80ebd44f27899f97211266c7b2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=87e8ca2152ac403cb71e5eb67c0b2e0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.339046 28352 tablet_service.cc:1467] Processing CreateTablet for tablet 585ba864cc384e7f9db90e9d4c47e82c (DEFAULT_TABLE table=rebalance_test_table_6 [id=87e8ca2152ac403cb71e5eb67c0b2e0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.339228 28483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3b8b80ebd44f27899f97211266c7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.339434 28352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585ba864cc384e7f9db90e9d4c47e82c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.341300 28353 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3b8b80ebd44f27899f97211266c7b2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=87e8ca2152ac403cb71e5eb67c0b2e0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.341650 28353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3b8b80ebd44f27899f97211266c7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.347584 28615 tablet_service.cc:1467] Processing CreateTablet for tablet 585ba864cc384e7f9db90e9d4c47e82c (DEFAULT_TABLE table=rebalance_test_table_6 [id=87e8ca2152ac403cb71e5eb67c0b2e0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.348009 29872 tablet_bootstrap.cc:492] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:49.348094 28615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585ba864cc384e7f9db90e9d4c47e82c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.350257 29872 tablet_bootstrap.cc:654] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.354584 29875 tablet_bootstrap.cc:492] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:49.355501 28745 tablet_service.cc:1467] Processing CreateTablet for tablet 585ba864cc384e7f9db90e9d4c47e82c (DEFAULT_TABLE table=rebalance_test_table_6 [id=87e8ca2152ac403cb71e5eb67c0b2e0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.356060 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585ba864cc384e7f9db90e9d4c47e82c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.356729 28744 tablet_service.cc:1467] Processing CreateTablet for tablet a20489bf65774902af38d228ee966ffe (DEFAULT_TABLE table=rebalance_test_table_6 [id=87e8ca2152ac403cb71e5eb67c0b2e0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.357134 28744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20489bf65774902af38d228ee966ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.357231 28483 tablet_service.cc:1467] Processing CreateTablet for tablet a20489bf65774902af38d228ee966ffe (DEFAULT_TABLE table=rebalance_test_table_6 [id=87e8ca2152ac403cb71e5eb67c0b2e0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.357651 28483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20489bf65774902af38d228ee966ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.366096 28354 tablet_service.cc:1467] Processing CreateTablet for tablet a20489bf65774902af38d228ee966ffe (DEFAULT_TABLE table=rebalance_test_table_6 [id=87e8ca2152ac403cb71e5eb67c0b2e0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.366577 28354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20489bf65774902af38d228ee966ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.366571 29875 tablet_bootstrap.cc:654] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.369529 29872 tablet_bootstrap.cc:492] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:49.369745 29872 ts_tablet_manager.cc:1367] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.017s
I20240503 08:12:49.370821 29872 raft_consensus.cc:348] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:49.371109 29872 raft_consensus.cc:374] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.371196 29872 raft_consensus.cc:729] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.371522 29872 consensus_queue.cc:260] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:49.372406 29872 ts_tablet_manager.cc:1398] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 08:12:49.373281 29872 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:49.373634 29875 tablet_bootstrap.cc:492] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:49.373958 29875 ts_tablet_manager.cc:1367] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.005s
I20240503 08:12:49.374910 29875 raft_consensus.cc:348] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.375196 29875 raft_consensus.cc:374] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.375346 29875 raft_consensus.cc:729] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.376838 29875 consensus_queue.cc:260] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.377655 29875 ts_tablet_manager.cc:1398] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:49.381330 29872 tablet_bootstrap.cc:654] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.388435 29877 tablet_bootstrap.cc:492] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:49.390101 29872 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:49.390339 29872 ts_tablet_manager.cc:1367] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.009s
I20240503 08:12:49.390729 29877 tablet_bootstrap.cc:654] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.391549 29872 raft_consensus.cc:348] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.391880 29872 raft_consensus.cc:374] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.392046 29872 raft_consensus.cc:729] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.392333 29872 consensus_queue.cc:260] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.392930 29872 ts_tablet_manager.cc:1398] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240503 08:12:49.393734 29877 tablet_bootstrap.cc:492] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:49.392454 29873 tablet_bootstrap.cc:492] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:49.393900 29877 ts_tablet_manager.cc:1367] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:12:49.394731 29877 raft_consensus.cc:348] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:49.394977 29877 raft_consensus.cc:374] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.395073 29877 raft_consensus.cc:729] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.395279 29873 tablet_bootstrap.cc:654] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.395329 29877 consensus_queue.cc:260] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:49.397215 29873 tablet_bootstrap.cc:492] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:49.397322 29877 ts_tablet_manager.cc:1398] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:49.397375 29873 ts_tablet_manager.cc:1367] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:12:49.397624 29877 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:49.398072 29873 raft_consensus.cc:348] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:49.398295 29873 raft_consensus.cc:374] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.398401 29873 raft_consensus.cc:729] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.398634 29873 consensus_queue.cc:260] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:49.399158 29873 ts_tablet_manager.cc:1398] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:49.399665 29877 tablet_bootstrap.cc:654] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.399801 29873 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:49.401371 29873 tablet_bootstrap.cc:654] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.403168 29877 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:49.403335 29877 ts_tablet_manager.cc:1367] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:49.404230 29873 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:49.404088 29877 raft_consensus.cc:348] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.404359 29877 raft_consensus.cc:374] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.404399 29873 ts_tablet_manager.cc:1367] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:12:49.404438 29877 raft_consensus.cc:729] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.404726 29877 consensus_queue.cc:260] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.405153 29873 raft_consensus.cc:348] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.405375 29873 raft_consensus.cc:374] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.405411 29877 ts_tablet_manager.cc:1398] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:49.405458 29873 raft_consensus.cc:729] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.405790 29877 tablet_bootstrap.cc:492] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:49.405879 29873 consensus_queue.cc:260] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.406404 29873 ts_tablet_manager.cc:1398] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:49.406873 29873 tablet_bootstrap.cc:492] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:49.407608 29877 tablet_bootstrap.cc:654] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.408162 29873 tablet_bootstrap.cc:654] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.412426 29859 raft_consensus.cc:482] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:49.412506 29873 tablet_bootstrap.cc:492] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:49.412664 29873 ts_tablet_manager.cc:1367] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 08:12:49.412645 29859 raft_consensus.cc:504] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.412940 29877 tablet_bootstrap.cc:492] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:49.413106 29877 ts_tablet_manager.cc:1367] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 08:12:49.413463 29873 raft_consensus.cc:348] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.413584 29859 leader_election.cc:288] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:49.413676 29873 raft_consensus.cc:374] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.413782 29873 raft_consensus.cc:729] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.414021 29873 consensus_queue.cc:260] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.414003 29877 raft_consensus.cc:348] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.414192 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585ba864cc384e7f9db90e9d4c47e82c" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" is_pre_election: true
I20240503 08:12:49.414238 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585ba864cc384e7f9db90e9d4c47e82c" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:49.414247 29877 raft_consensus.cc:374] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.414449 28765 raft_consensus.cc:2454] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 0.
I20240503 08:12:49.414489 29877 raft_consensus.cc:729] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.414597 29873 ts_tablet_manager.cc:1398] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:49.414769 29877 consensus_queue.cc:260] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.415088 28374 raft_consensus.cc:2454] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 0.
I20240503 08:12:49.415375 29877 ts_tablet_manager.cc:1398] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:49.415711 28569 leader_election.cc:302] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:49.416232 29859 raft_consensus.cc:2781] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:49.416370 29859 raft_consensus.cc:482] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:49.416474 29859 raft_consensus.cc:3037] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.419334 29859 raft_consensus.cc:504] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.419982 29859 leader_election.cc:288] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 election: Requested vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:49.420390 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585ba864cc384e7f9db90e9d4c47e82c" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:12:49.420472 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585ba864cc384e7f9db90e9d4c47e82c" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:49.420603 28374 raft_consensus.cc:3037] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.420683 28765 raft_consensus.cc:3037] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.423607 28374 raft_consensus.cc:2454] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 1.
I20240503 08:12:49.423607 28765 raft_consensus.cc:2454] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 1.
I20240503 08:12:49.424042 28569 leader_election.cc:302] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:49.424408 29859 raft_consensus.cc:2781] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:49.424571 29859 raft_consensus.cc:686] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Becoming Leader. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Running, Role: LEADER
I20240503 08:12:49.425097 29859 consensus_queue.cc:237] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.428166 28231 catalog_manager.cc:5484] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4574396f35514c9caccb346f1a7af84d (127.20.107.131). New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:49.461042 29555 raft_consensus.cc:482] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:49.461249 29555 raft_consensus.cc:504] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.462014 29555 leader_election.cc:288] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:49.462587 28373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20489bf65774902af38d228ee966ffe" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" is_pre_election: true
I20240503 08:12:49.462617 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20489bf65774902af38d228ee966ffe" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:49.462805 28373 raft_consensus.cc:2454] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:49.462883 28765 raft_consensus.cc:2454] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:49.463320 28439 leader_election.cc:302] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:49.463637 29555 raft_consensus.cc:2781] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:49.463748 29555 raft_consensus.cc:482] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:49.463876 29555 raft_consensus.cc:3037] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.465798 29555 raft_consensus.cc:504] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.466396 29555 leader_election.cc:288] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Requested vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:49.466969 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20489bf65774902af38d228ee966ffe" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:49.467002 28373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20489bf65774902af38d228ee966ffe" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:12:49.467170 28373 raft_consensus.cc:3037] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.467193 28765 raft_consensus.cc:3037] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.469264 28373 raft_consensus.cc:2454] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:49.469318 28765 raft_consensus.cc:2454] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:49.469794 28439 leader_election.cc:302] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716ffbb2a00b4f43b03e5cedcc9d302a, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:49.470069 29555 raft_consensus.cc:2781] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:49.470259 29555 raft_consensus.cc:686] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Becoming Leader. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Running, Role: LEADER
I20240503 08:12:49.470538 29555 consensus_queue.cc:237] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:49.473634 28231 catalog_manager.cc:5484] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: term changed from 0 to 1, leader changed from <none> to 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130). New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:49.687227 29867 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:49.687436 29867 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:49.688128 29867 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:49.688673 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3b8b80ebd44f27899f97211266c7b2" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:49.688680 28505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3b8b80ebd44f27899f97211266c7b2" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:49.688949 28765 raft_consensus.cc:2454] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:49.688963 28505 raft_consensus.cc:2454] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:49.689411 28308 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:49.689683 29867 raft_consensus.cc:2781] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:49.689837 29867 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:49.689941 29867 raft_consensus.cc:3037] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.692126 29867 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:49.692699 29867 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Requested vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:49.693255 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3b8b80ebd44f27899f97211266c7b2" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:49.693269 28505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3b8b80ebd44f27899f97211266c7b2" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:12:49.693500 28765 raft_consensus.cc:3037] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.693588 28505 raft_consensus.cc:3037] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.695523 28505 raft_consensus.cc:2454] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:49.695540 28765 raft_consensus.cc:2454] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:49.695995 28308 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:49.696314 29867 raft_consensus.cc:2781] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:49.696467 29867 raft_consensus.cc:686] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Becoming Leader. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Running, Role: LEADER
I20240503 08:12:49.696779 29867 consensus_queue.cc:237] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:49.700078 28231 catalog_manager.cc:5484] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129). New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:49.836939 29859 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.843786 29754 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.895658 29555 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.901536 29555 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:50.088371 29888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:50.088759 29888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:50.097191 29888 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 08:12:50.251143 29716 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:50.287211 29716 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 317.623 ms
time per row: 0.158812 ms
I20240503 08:12:50.572957 28231 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60904:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:50.582087 28353 tablet_service.cc:1467] Processing CreateTablet for tablet b4ceeeb3495e4a8fac6e13e431c64af4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0225804a5c4a42cbb32c4b0029752bd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.582435 28615 tablet_service.cc:1467] Processing CreateTablet for tablet b4ceeeb3495e4a8fac6e13e431c64af4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0225804a5c4a42cbb32c4b0029752bd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.582590 28353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ceeeb3495e4a8fac6e13e431c64af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.582893 28615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ceeeb3495e4a8fac6e13e431c64af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.582903 28743 tablet_service.cc:1467] Processing CreateTablet for tablet 564fb1b51abd4f8da0372ec99e3c599c (DEFAULT_TABLE table=rebalance_test_table_7 [id=0225804a5c4a42cbb32c4b0029752bd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.583302 28743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564fb1b51abd4f8da0372ec99e3c599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.583664 28483 tablet_service.cc:1467] Processing CreateTablet for tablet b4ceeeb3495e4a8fac6e13e431c64af4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0225804a5c4a42cbb32c4b0029752bd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.584053 28483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ceeeb3495e4a8fac6e13e431c64af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.589738 28614 tablet_service.cc:1467] Processing CreateTablet for tablet 564fb1b51abd4f8da0372ec99e3c599c (DEFAULT_TABLE table=rebalance_test_table_7 [id=0225804a5c4a42cbb32c4b0029752bd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.590162 28614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564fb1b51abd4f8da0372ec99e3c599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.590242 28744 tablet_service.cc:1467] Processing CreateTablet for tablet b5736ee8065344ea8124fdf759035216 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0225804a5c4a42cbb32c4b0029752bd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.590600 28744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5736ee8065344ea8124fdf759035216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.590791 28485 tablet_service.cc:1467] Processing CreateTablet for tablet b5736ee8065344ea8124fdf759035216 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0225804a5c4a42cbb32c4b0029752bd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.591153 28485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5736ee8065344ea8124fdf759035216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.593241 28613 tablet_service.cc:1467] Processing CreateTablet for tablet b5736ee8065344ea8124fdf759035216 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0225804a5c4a42cbb32c4b0029752bd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.593242 28352 tablet_service.cc:1467] Processing CreateTablet for tablet 564fb1b51abd4f8da0372ec99e3c599c (DEFAULT_TABLE table=rebalance_test_table_7 [id=0225804a5c4a42cbb32c4b0029752bd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.593588 28613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5736ee8065344ea8124fdf759035216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.593601 28352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564fb1b51abd4f8da0372ec99e3c599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.595541 29954 tablet_bootstrap.cc:492] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:50.597326 29954 tablet_bootstrap.cc:654] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.608350 29957 tablet_bootstrap.cc:492] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:50.608693 29955 tablet_bootstrap.cc:492] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:50.610493 29957 tablet_bootstrap.cc:654] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.610527 29955 tablet_bootstrap.cc:654] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.611163 29953 tablet_bootstrap.cc:492] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:50.611901 29954 tablet_bootstrap.cc:492] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:50.612118 29954 ts_tablet_manager.cc:1367] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240503 08:12:50.612802 29954 raft_consensus.cc:348] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.613045 29954 raft_consensus.cc:374] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.613154 29954 raft_consensus.cc:729] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.613224 29953 tablet_bootstrap.cc:654] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.613363 29955 tablet_bootstrap.cc:492] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:50.613507 29955 ts_tablet_manager.cc:1367] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 08:12:50.613487 29954 consensus_queue.cc:260] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.614336 29955 raft_consensus.cc:348] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:50.614634 29955 raft_consensus.cc:374] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.614728 29955 raft_consensus.cc:729] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.614976 29955 consensus_queue.cc:260] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:50.615396 29954 ts_tablet_manager.cc:1398] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:50.615458 29957 tablet_bootstrap.cc:492] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:50.615612 29957 ts_tablet_manager.cc:1367] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240503 08:12:50.615957 29955 ts_tablet_manager.cc:1398] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:50.616230 29955 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:50.616413 29957 raft_consensus.cc:348] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.616595 29957 raft_consensus.cc:374] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.616677 29957 raft_consensus.cc:729] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.616935 29957 consensus_queue.cc:260] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.617776 29953 tablet_bootstrap.cc:492] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:50.617908 29955 tablet_bootstrap.cc:654] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.617995 29953 ts_tablet_manager.cc:1367] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 08:12:50.618839 29953 raft_consensus.cc:348] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.619201 29953 raft_consensus.cc:374] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.619334 29953 raft_consensus.cc:729] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.619900 29953 consensus_queue.cc:260] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.620596 29957 ts_tablet_manager.cc:1398] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:50.621384 29954 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:50.622864 29954 tablet_bootstrap.cc:654] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.626420 29953 ts_tablet_manager.cc:1398] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:12:50.626758 29953 tablet_bootstrap.cc:492] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:50.626830 29955 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:50.627086 29955 ts_tablet_manager.cc:1367] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240503 08:12:50.628047 29957 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:50.628109 29955 raft_consensus.cc:348] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.628465 29955 raft_consensus.cc:374] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.628592 29955 raft_consensus.cc:729] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.629029 29955 consensus_queue.cc:260] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.629834 29957 tablet_bootstrap.cc:654] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.631086 29955 ts_tablet_manager.cc:1398] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:50.631461 29953 tablet_bootstrap.cc:654] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.633860 29957 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:50.634054 29957 ts_tablet_manager.cc:1367] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 08:12:50.634877 29957 raft_consensus.cc:348] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.635169 29957 raft_consensus.cc:374] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.635249 29957 raft_consensus.cc:729] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.635525 29953 tablet_bootstrap.cc:492] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:50.635552 29957 consensus_queue.cc:260] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.635749 29953 ts_tablet_manager.cc:1367] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 08:12:50.636363 29957 ts_tablet_manager.cc:1398] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 08:12:50.636974 29953 raft_consensus.cc:348] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:50.637187 29953 raft_consensus.cc:374] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.637267 29953 raft_consensus.cc:729] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.637465 29954 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:50.637523 29953 consensus_queue.cc:260] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:50.637773 29954 ts_tablet_manager.cc:1367] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20240503 08:12:50.638612 29953 ts_tablet_manager.cc:1398] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:50.638511 29954 raft_consensus.cc:348] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.638749 29954 raft_consensus.cc:374] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.638854 29954 raft_consensus.cc:729] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.639158 29954 consensus_queue.cc:260] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.640470 29954 ts_tablet_manager.cc:1398] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:50.640740 29954 tablet_bootstrap.cc:492] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:50.642514 29954 tablet_bootstrap.cc:654] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.644706 29954 tablet_bootstrap.cc:492] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:50.644903 29954 ts_tablet_manager.cc:1367] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:50.645839 29954 raft_consensus.cc:348] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:50.646083 29954 raft_consensus.cc:374] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.646167 29954 raft_consensus.cc:729] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.646373 29954 consensus_queue.cc:260] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:50.646876 29954 ts_tablet_manager.cc:1398] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:50.663075 29859 raft_consensus.cc:482] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:50.663237 29859 raft_consensus.cc:504] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.663823 29859 leader_election.cc:288] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:50.664238 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ceeeb3495e4a8fac6e13e431c64af4" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" is_pre_election: true
I20240503 08:12:50.664327 28505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ceeeb3495e4a8fac6e13e431c64af4" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:50.664461 28374 raft_consensus.cc:2454] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 0.
I20240503 08:12:50.664532 28505 raft_consensus.cc:2454] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 0.
I20240503 08:12:50.664923 28569 leader_election.cc:302] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 4e4a1242f3714d03a6c9ee51dd63929c; no voters:
I20240503 08:12:50.665232 29859 raft_consensus.cc:2781] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:50.665374 29859 raft_consensus.cc:482] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:50.665483 29859 raft_consensus.cc:3037] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.667467 29859 raft_consensus.cc:504] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.667991 29859 leader_election.cc:288] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 election: Requested vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:50.668473 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ceeeb3495e4a8fac6e13e431c64af4" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:12:50.668567 28505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ceeeb3495e4a8fac6e13e431c64af4" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:12:50.668696 28374 raft_consensus.cc:3037] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.668787 28505 raft_consensus.cc:3037] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.670768 28505 raft_consensus.cc:2454] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 1.
I20240503 08:12:50.670776 28374 raft_consensus.cc:2454] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 1.
I20240503 08:12:50.671229 28569 leader_election.cc:302] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 4e4a1242f3714d03a6c9ee51dd63929c; no voters:
I20240503 08:12:50.671538 29859 raft_consensus.cc:2781] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:50.671700 29859 raft_consensus.cc:686] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Becoming Leader. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Running, Role: LEADER
I20240503 08:12:50.671973 29859 consensus_queue.cc:237] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.675328 28231 catalog_manager.cc:5484] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4574396f35514c9caccb346f1a7af84d (127.20.107.131). New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:50.834041 29549 raft_consensus.cc:482] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:50.834271 29549 raft_consensus.cc:504] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.834961 29549 leader_election.cc:288] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:50.835566 28635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5736ee8065344ea8124fdf759035216" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
I20240503 08:12:50.835611 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5736ee8065344ea8124fdf759035216" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:50.835775 28635 raft_consensus.cc:2454] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:50.835935 28764 raft_consensus.cc:2454] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:50.836263 28438 leader_election.cc:302] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:50.836546 29549 raft_consensus.cc:2781] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:50.836632 29549 raft_consensus.cc:482] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:50.836740 29549 raft_consensus.cc:3037] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.838909 29549 raft_consensus.cc:504] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.840145 29549 leader_election.cc:288] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Requested vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:50.840730 28635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5736ee8065344ea8124fdf759035216" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:50.840730 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5736ee8065344ea8124fdf759035216" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:50.840975 28635 raft_consensus.cc:3037] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.841001 28764 raft_consensus.cc:3037] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.843786 28764 raft_consensus.cc:2454] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:50.844300 28439 leader_election.cc:302] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716ffbb2a00b4f43b03e5cedcc9d302a, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:50.844429 28635 raft_consensus.cc:2454] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:50.844655 29549 raft_consensus.cc:2781] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:50.844976 29549 raft_consensus.cc:686] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Becoming Leader. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Running, Role: LEADER
I20240503 08:12:50.845268 29549 consensus_queue.cc:237] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:50.848575 28231 catalog_manager.cc:5484] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: term changed from 0 to 1, leader changed from <none> to 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130). New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:51.048332 29803 raft_consensus.cc:482] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:51.048558 29803 raft_consensus.cc:504] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:51.049242 29803 leader_election.cc:288] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:12:51.049835 28635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564fb1b51abd4f8da0372ec99e3c599c" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
I20240503 08:12:51.049849 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564fb1b51abd4f8da0372ec99e3c599c" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" is_pre_election: true
I20240503 08:12:51.050061 28374 raft_consensus.cc:2454] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 0.
I20240503 08:12:51.050053 28635 raft_consensus.cc:2454] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 0.
I20240503 08:12:51.050460 28698 leader_election.cc:302] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:51.050994 29803 raft_consensus.cc:2781] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:51.051106 29803 raft_consensus.cc:482] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:51.051219 29803 raft_consensus.cc:3037] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.053206 29803 raft_consensus.cc:504] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:51.053773 29803 leader_election.cc:288] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 election: Requested vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:12:51.054203 28635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564fb1b51abd4f8da0372ec99e3c599c" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:51.054260 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564fb1b51abd4f8da0372ec99e3c599c" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:12:51.054394 28635 raft_consensus.cc:3037] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.054478 28374 raft_consensus.cc:3037] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.056526 28635 raft_consensus.cc:2454] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 1.
I20240503 08:12:51.056619 28374 raft_consensus.cc:2454] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 1.
I20240503 08:12:51.057093 28698 leader_election.cc:302] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:51.057396 29803 raft_consensus.cc:2781] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:51.057566 29803 raft_consensus.cc:686] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Becoming Leader. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Running, Role: LEADER
I20240503 08:12:51.057840 29803 consensus_queue.cc:237] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:51.060801 28231 catalog_manager.cc:5484] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132). New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:51.073591 29859 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.080405 29819 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.445339 29549 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.464604 29981 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.524030 29803 consensus_queue.cc:1035] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.529484 29968 consensus_queue.cc:1035] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:51.558352 29977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:51.558699 29977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:51.566792 29977 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
INSERT report
rows total: 2000
time total: 303.678 ms
time per row: 0.151839 ms
I20240503 08:12:52.030272 28230 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60904:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:52.039757 28353 tablet_service.cc:1467] Processing CreateTablet for tablet ba833ce23c4c4e81bc1a81125a1208c9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=649380be96b24c78aa15b40471703d93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.040299 28353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba833ce23c4c4e81bc1a81125a1208c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.040263 28483 tablet_service.cc:1467] Processing CreateTablet for tablet ba833ce23c4c4e81bc1a81125a1208c9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=649380be96b24c78aa15b40471703d93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.040756 28483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba833ce23c4c4e81bc1a81125a1208c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.041234 28613 tablet_service.cc:1467] Processing CreateTablet for tablet 2d113e8de02f447b9da0a1cb03686cd2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=649380be96b24c78aa15b40471703d93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.041673 28613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d113e8de02f447b9da0a1cb03686cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.041610 28743 tablet_service.cc:1467] Processing CreateTablet for tablet a8a9adc3712d436c95505a3216e7607c (DEFAULT_TABLE table=rebalance_test_table_8 [id=649380be96b24c78aa15b40471703d93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.041893 28352 tablet_service.cc:1467] Processing CreateTablet for tablet a8a9adc3712d436c95505a3216e7607c (DEFAULT_TABLE table=rebalance_test_table_8 [id=649380be96b24c78aa15b40471703d93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.042028 28743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a9adc3712d436c95505a3216e7607c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.042320 28352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a9adc3712d436c95505a3216e7607c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.045333 28744 tablet_service.cc:1467] Processing CreateTablet for tablet 2d113e8de02f447b9da0a1cb03686cd2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=649380be96b24c78aa15b40471703d93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.045789 28744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d113e8de02f447b9da0a1cb03686cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.048496 28615 tablet_service.cc:1467] Processing CreateTablet for tablet ba833ce23c4c4e81bc1a81125a1208c9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=649380be96b24c78aa15b40471703d93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.048655 30042 tablet_bootstrap.cc:492] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:52.049741 28615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba833ce23c4c4e81bc1a81125a1208c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.049909 28485 tablet_service.cc:1467] Processing CreateTablet for tablet a8a9adc3712d436c95505a3216e7607c (DEFAULT_TABLE table=rebalance_test_table_8 [id=649380be96b24c78aa15b40471703d93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.050312 28485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a9adc3712d436c95505a3216e7607c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.058045 30043 tablet_bootstrap.cc:492] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:52.058671 28354 tablet_service.cc:1467] Processing CreateTablet for tablet 2d113e8de02f447b9da0a1cb03686cd2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=649380be96b24c78aa15b40471703d93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.059141 28354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d113e8de02f447b9da0a1cb03686cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.060266 30043 tablet_bootstrap.cc:654] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.062000 30044 tablet_bootstrap.cc:492] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:52.064091 30044 tablet_bootstrap.cc:654] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.067495 30041 tablet_bootstrap.cc:492] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:52.069499 30041 tablet_bootstrap.cc:654] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.070436 30043 tablet_bootstrap.cc:492] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:52.070657 30042 tablet_bootstrap.cc:654] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.070732 30043 ts_tablet_manager.cc:1367] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240503 08:12:52.070842 30044 tablet_bootstrap.cc:492] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:52.071211 30044 ts_tablet_manager.cc:1367] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20240503 08:12:52.071946 30043 raft_consensus.cc:348] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.072291 30043 raft_consensus.cc:374] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.072134 30044 raft_consensus.cc:348] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.072450 30043 raft_consensus.cc:729] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.072503 30044 raft_consensus.cc:374] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.072669 30044 raft_consensus.cc:729] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.072901 30043 consensus_queue.cc:260] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.073042 30044 consensus_queue.cc:260] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.077330 30043 ts_tablet_manager.cc:1398] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:52.078097 30044 ts_tablet_manager.cc:1398] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:12:52.078959 30043 tablet_bootstrap.cc:492] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:52.081650 30042 tablet_bootstrap.cc:492] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:52.081885 30042 ts_tablet_manager.cc:1367] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.004s
I20240503 08:12:52.081934 30043 tablet_bootstrap.cc:654] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.082857 30044 tablet_bootstrap.cc:492] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:52.082798 30042 raft_consensus.cc:348] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:52.083077 30042 raft_consensus.cc:374] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.083213 30042 raft_consensus.cc:729] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.083506 30042 consensus_queue.cc:260] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:52.084570 30044 tablet_bootstrap.cc:654] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.087064 30042 ts_tablet_manager.cc:1398] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:12:52.087450 30042 tablet_bootstrap.cc:492] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:52.090700 30042 tablet_bootstrap.cc:654] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.090972 30043 tablet_bootstrap.cc:492] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:52.091167 30043 ts_tablet_manager.cc:1367] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240503 08:12:52.092336 30043 raft_consensus.cc:348] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:52.092711 30043 raft_consensus.cc:374] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.092860 30043 raft_consensus.cc:729] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.093297 30043 consensus_queue.cc:260] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:52.093705 30041 tablet_bootstrap.cc:492] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:52.093860 30041 ts_tablet_manager.cc:1367] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.015s
I20240503 08:12:52.094485 30041 raft_consensus.cc:348] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.094660 30041 raft_consensus.cc:374] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.094750 30041 raft_consensus.cc:729] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.094976 30041 consensus_queue.cc:260] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.095562 30041 ts_tablet_manager.cc:1398] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:52.096029 30041 tablet_bootstrap.cc:492] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:52.096062 30044 tablet_bootstrap.cc:492] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:52.096251 30044 ts_tablet_manager.cc:1367] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20240503 08:12:52.096350 30043 ts_tablet_manager.cc:1398] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:52.097030 30044 raft_consensus.cc:348] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.097275 30044 raft_consensus.cc:374] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.097357 30044 raft_consensus.cc:729] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.097757 30041 tablet_bootstrap.cc:654] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.097652 30044 consensus_queue.cc:260] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.098402 30044 ts_tablet_manager.cc:1398] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:52.099315 30042 tablet_bootstrap.cc:492] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:52.099539 30042 ts_tablet_manager.cc:1367] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240503 08:12:52.100283 30042 raft_consensus.cc:348] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.100459 30042 raft_consensus.cc:374] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.100558 30042 raft_consensus.cc:729] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.100842 30042 consensus_queue.cc:260] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.101042 30041 tablet_bootstrap.cc:492] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:52.101478 30042 ts_tablet_manager.cc:1398] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:52.102020 30041 ts_tablet_manager.cc:1367] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:12:52.102865 30041 raft_consensus.cc:348] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.103057 30041 raft_consensus.cc:374] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.103193 30041 raft_consensus.cc:729] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.103441 30041 consensus_queue.cc:260] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.103952 30041 ts_tablet_manager.cc:1398] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:52.104513 30041 tablet_bootstrap.cc:492] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:52.106200 30041 tablet_bootstrap.cc:654] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.107898 30041 tablet_bootstrap.cc:492] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:52.108016 30041 ts_tablet_manager.cc:1367] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:52.108690 30041 raft_consensus.cc:348] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:52.108834 30041 raft_consensus.cc:374] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.108927 30041 raft_consensus.cc:729] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.109197 30041 consensus_queue.cc:260] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:52.109604 30041 ts_tablet_manager.cc:1398] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:52.117090 30037 raft_consensus.cc:482] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:52.117252 30037 raft_consensus.cc:504] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.117846 30037 leader_election.cc:288] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:52.118386 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9adc3712d436c95505a3216e7607c" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:52.118441 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9adc3712d436c95505a3216e7607c" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:52.118628 28764 raft_consensus.cc:2454] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:52.118675 28504 raft_consensus.cc:2454] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:52.119043 28308 leader_election.cc:302] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:52.119295 30037 raft_consensus.cc:2781] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:52.119417 30037 raft_consensus.cc:482] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:52.119510 30037 raft_consensus.cc:3037] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.121572 30037 raft_consensus.cc:504] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.122118 30037 leader_election.cc:288] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Requested vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:52.122447 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9adc3712d436c95505a3216e7607c" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:52.122651 28764 raft_consensus.cc:3037] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.122586 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9adc3712d436c95505a3216e7607c" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:12:52.122797 28504 raft_consensus.cc:3037] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.124763 28764 raft_consensus.cc:2454] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:52.124785 28504 raft_consensus.cc:2454] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:52.125334 28308 leader_election.cc:302] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:52.125701 30037 raft_consensus.cc:2781] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:52.125870 30037 raft_consensus.cc:686] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Becoming Leader. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Running, Role: LEADER
I20240503 08:12:52.126147 30037 consensus_queue.cc:237] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.129099 28231 catalog_manager.cc:5484] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129). New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:52.204584 30037 raft_consensus.cc:482] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:52.204852 30037 raft_consensus.cc:504] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.205098 29819 raft_consensus.cc:482] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:52.205293 29819 raft_consensus.cc:504] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:52.206192 29819 leader_election.cc:288] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:12:52.206918 30037 leader_election.cc:288] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:52.207531 28635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba833ce23c4c4e81bc1a81125a1208c9" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
I20240503 08:12:52.207549 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba833ce23c4c4e81bc1a81125a1208c9" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:52.206758 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d113e8de02f447b9da0a1cb03686cd2" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:52.207727 28635 raft_consensus.cc:2454] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:52.207791 28504 raft_consensus.cc:2454] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:52.207793 28764 raft_consensus.cc:2454] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 0.
I20240503 08:12:52.206845 28373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d113e8de02f447b9da0a1cb03686cd2" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" is_pre_election: true
I20240503 08:12:52.208065 28373 raft_consensus.cc:2454] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 0.
I20240503 08:12:52.208309 28308 leader_election.cc:302] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:52.208652 30037 raft_consensus.cc:2781] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:52.208767 30037 raft_consensus.cc:482] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:52.208760 28569 leader_election.cc:302] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 4e4a1242f3714d03a6c9ee51dd63929c; no voters:
I20240503 08:12:52.208894 30037 raft_consensus.cc:3037] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.209321 29819 raft_consensus.cc:2781] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:52.209430 29819 raft_consensus.cc:482] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:52.209542 29819 raft_consensus.cc:3037] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.212085 30037 raft_consensus.cc:504] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.212632 29819 raft_consensus.cc:504] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:52.213428 29819 leader_election.cc:288] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 election: Requested vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:12:52.213395 28635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba833ce23c4c4e81bc1a81125a1208c9" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:52.213428 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba833ce23c4c4e81bc1a81125a1208c9" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:12:52.213613 28635 raft_consensus.cc:3037] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.213634 28504 raft_consensus.cc:3037] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.213663 30037 leader_election.cc:288] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Requested vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:12:52.214360 28764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d113e8de02f447b9da0a1cb03686cd2" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:52.214576 28764 raft_consensus.cc:3037] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.216325 28635 raft_consensus.cc:2454] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:52.216379 28504 raft_consensus.cc:2454] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:52.216902 28373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d113e8de02f447b9da0a1cb03686cd2" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:12:52.217106 28373 raft_consensus.cc:3037] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.217552 28764 raft_consensus.cc:2454] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 1.
I20240503 08:12:52.217736 28308 leader_election.cc:302] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:52.218156 30037 raft_consensus.cc:2781] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:52.218323 30037 raft_consensus.cc:686] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Becoming Leader. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Running, Role: LEADER
I20240503 08:12:52.218600 30037 consensus_queue.cc:237] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:12:52.219725 28569 leader_election.cc:302] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:52.220060 28373 raft_consensus.cc:2454] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 1.
I20240503 08:12:52.220216 29819 raft_consensus.cc:2781] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:52.220906 29819 raft_consensus.cc:686] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Becoming Leader. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Running, Role: LEADER
I20240503 08:12:52.221241 29819 consensus_queue.cc:237] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } }
I20240503 08:12:52.224144 28231 catalog_manager.cc:5484] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129). New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:52.224747 28230 catalog_manager.cc:5484] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4574396f35514c9caccb346f1a7af84d (127.20.107.131). New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:52.579804 30037 consensus_queue.cc:1035] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.590597 29716 consensus_queue.cc:1035] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:52.650208 30058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:52.650525 30058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:52.658142 30058 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 08:12:52.711030 29819 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.718461 29819 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.775835 30059 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.795984 29716 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:52.866060 28699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.130:40397, user_credentials={real_user=slave}} blocked reactor thread for 51955.6us
INSERT report
rows total: 2000
time total: 306.406 ms
time per row: 0.153203 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:12:54.118162 28227 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60904:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:54.128046 28744 tablet_service.cc:1467] Processing CreateTablet for tablet 6e9537f30aaf45adbc2cc4a784009de7 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e4d3a510e86a4cbe8d3d611eac908187]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.128558 28744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9537f30aaf45adbc2cc4a784009de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.128515 28615 tablet_service.cc:1467] Processing CreateTablet for tablet 6e9537f30aaf45adbc2cc4a784009de7 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e4d3a510e86a4cbe8d3d611eac908187]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.128701 28483 tablet_service.cc:1467] Processing CreateTablet for tablet da529c8f8de241d38678d1596b9402ac (DEFAULT_TABLE table=rebalance_test_table_9 [id=e4d3a510e86a4cbe8d3d611eac908187]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.128937 28615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9537f30aaf45adbc2cc4a784009de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.129132 28483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da529c8f8de241d38678d1596b9402ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.130442 28353 tablet_service.cc:1467] Processing CreateTablet for tablet 48e31346107a453a81921b23b244e7ff (DEFAULT_TABLE table=rebalance_test_table_9 [id=e4d3a510e86a4cbe8d3d611eac908187]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.130852 28353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e31346107a453a81921b23b244e7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.133268 28613 tablet_service.cc:1467] Processing CreateTablet for tablet da529c8f8de241d38678d1596b9402ac (DEFAULT_TABLE table=rebalance_test_table_9 [id=e4d3a510e86a4cbe8d3d611eac908187]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.133670 28613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da529c8f8de241d38678d1596b9402ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.134856 28743 tablet_service.cc:1467] Processing CreateTablet for tablet da529c8f8de241d38678d1596b9402ac (DEFAULT_TABLE table=rebalance_test_table_9 [id=e4d3a510e86a4cbe8d3d611eac908187]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.135308 28743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da529c8f8de241d38678d1596b9402ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.138278 28614 tablet_service.cc:1467] Processing CreateTablet for tablet 48e31346107a453a81921b23b244e7ff (DEFAULT_TABLE table=rebalance_test_table_9 [id=e4d3a510e86a4cbe8d3d611eac908187]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.138854 28614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e31346107a453a81921b23b244e7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.141180 28485 tablet_service.cc:1467] Processing CreateTablet for tablet 6e9537f30aaf45adbc2cc4a784009de7 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e4d3a510e86a4cbe8d3d611eac908187]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.141790 28485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9537f30aaf45adbc2cc4a784009de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.143839 30120 tablet_bootstrap.cc:492] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:54.145668 30120 tablet_bootstrap.cc:654] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.153924 30120 tablet_bootstrap.cc:492] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:54.154209 30120 ts_tablet_manager.cc:1367] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240503 08:12:54.155169 30120 raft_consensus.cc:348] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.155334 30120 raft_consensus.cc:374] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.155457 30120 raft_consensus.cc:729] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.155704 30120 consensus_queue.cc:260] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.156543 30120 ts_tablet_manager.cc:1398] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:54.156919 30120 tablet_bootstrap.cc:492] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:54.159022 30120 tablet_bootstrap.cc:654] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.159839 30124 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:12:54.164224 30124 tablet_bootstrap.cc:654] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.173494 30124 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c: No bootstrap required, opened a new log
I20240503 08:12:54.173828 30124 ts_tablet_manager.cc:1367] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240503 08:12:54.174897 30119 tablet_bootstrap.cc:492] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:54.174854 30124 raft_consensus.cc:348] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.175122 30124 raft_consensus.cc:374] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.175256 30124 raft_consensus.cc:729] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.175371 28484 tablet_service.cc:1467] Processing CreateTablet for tablet 48e31346107a453a81921b23b244e7ff (DEFAULT_TABLE table=rebalance_test_table_9 [id=e4d3a510e86a4cbe8d3d611eac908187]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.175578 30124 consensus_queue.cc:260] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.175900 28484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e31346107a453a81921b23b244e7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.176275 30124 ts_tablet_manager.cc:1398] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:54.177161 30119 tablet_bootstrap.cc:654] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.183941 30128 tablet_bootstrap.cc:492] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:54.185969 30128 tablet_bootstrap.cc:654] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.186450 30119 tablet_bootstrap.cc:492] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:54.186662 30119 ts_tablet_manager.cc:1367] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20240503 08:12:54.187503 30119 raft_consensus.cc:348] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.187778 30119 raft_consensus.cc:374] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.187896 30119 raft_consensus.cc:729] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.188477 30119 consensus_queue.cc:260] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.189085 30119 ts_tablet_manager.cc:1398] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:54.189405 30119 tablet_bootstrap.cc:492] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:12:54.190074 30120 tablet_bootstrap.cc:492] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:54.190258 30120 ts_tablet_manager.cc:1367] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.000s
I20240503 08:12:54.190982 30128 tablet_bootstrap.cc:492] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:54.191133 30128 ts_tablet_manager.cc:1367] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240503 08:12:54.191063 30120 raft_consensus.cc:348] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:54.191185 30119 tablet_bootstrap.cc:654] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.191282 30120 raft_consensus.cc:374] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.191381 30120 raft_consensus.cc:729] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.191617 30120 consensus_queue.cc:260] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:54.191972 30128 raft_consensus.cc:348] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:54.192152 30120 ts_tablet_manager.cc:1398] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:54.192157 30128 raft_consensus.cc:374] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.192466 30120 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:12:54.192239 30128 raft_consensus.cc:729] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.193802 30128 consensus_queue.cc:260] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:54.194176 30120 tablet_bootstrap.cc:654] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.199671 30119 tablet_bootstrap.cc:492] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb: No bootstrap required, opened a new log
I20240503 08:12:54.199910 30119 ts_tablet_manager.cc:1367] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240503 08:12:54.200701 30119 raft_consensus.cc:348] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:54.201135 30119 raft_consensus.cc:374] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.201244 30119 raft_consensus.cc:729] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.201521 30119 consensus_queue.cc:260] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:54.202244 30119 ts_tablet_manager.cc:1398] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:54.205010 30128 ts_tablet_manager.cc:1398] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240503 08:12:54.205462 30128 tablet_bootstrap.cc:492] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:54.206154 30120 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: No bootstrap required, opened a new log
I20240503 08:12:54.206350 30120 ts_tablet_manager.cc:1367] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.012s
I20240503 08:12:54.207116 30120 raft_consensus.cc:348] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.207357 30120 raft_consensus.cc:374] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.207466 30120 raft_consensus.cc:729] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.207734 30120 consensus_queue.cc:260] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.207872 30128 tablet_bootstrap.cc:654] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.208313 30120 ts_tablet_manager.cc:1398] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:54.218070 30128 tablet_bootstrap.cc:492] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:54.218267 30128 ts_tablet_manager.cc:1367] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20240503 08:12:54.219008 30128 raft_consensus.cc:348] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.219247 30128 raft_consensus.cc:374] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.219327 30128 raft_consensus.cc:729] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.219599 30128 consensus_queue.cc:260] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.220288 30128 ts_tablet_manager.cc:1398] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 08:12:54.220577 30128 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:12:54.222183 30128 tablet_bootstrap.cc:654] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.229668 30121 raft_consensus.cc:482] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:54.229851 30121 raft_consensus.cc:504] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:54.230226 30128 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: No bootstrap required, opened a new log
I20240503 08:12:54.230386 30128 ts_tablet_manager.cc:1367] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20240503 08:12:54.230723 30121 leader_election.cc:288] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:54.231269 30128 raft_consensus.cc:348] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.231444 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da529c8f8de241d38678d1596b9402ac" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb" is_pre_election: true
I20240503 08:12:54.231701 28765 raft_consensus.cc:2454] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:54.231652 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da529c8f8de241d38678d1596b9402ac" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
I20240503 08:12:54.231889 28634 raft_consensus.cc:2454] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 0.
I20240503 08:12:54.232175 30128 raft_consensus.cc:374] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.232303 28439 leader_election.cc:302] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716ffbb2a00b4f43b03e5cedcc9d302a, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:54.232329 30128 raft_consensus.cc:729] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.232635 30121 raft_consensus.cc:2781] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:54.232928 30121 raft_consensus.cc:482] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:54.233059 30121 raft_consensus.cc:3037] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.233162 30128 consensus_queue.cc:260] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.233870 30128 ts_tablet_manager.cc:1398] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:54.235234 30121 raft_consensus.cc:504] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:54.235857 30121 leader_election.cc:288] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Requested vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:12:54.236330 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da529c8f8de241d38678d1596b9402ac" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:54.236563 28634 raft_consensus.cc:3037] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.236572 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da529c8f8de241d38678d1596b9402ac" candidate_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:12:54.236836 28765 raft_consensus.cc:3037] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.239848 28634 raft_consensus.cc:2454] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:54.240360 28438 leader_election.cc:302] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:54.240763 30121 raft_consensus.cc:2781] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:54.241135 28765 raft_consensus.cc:2454] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716ffbb2a00b4f43b03e5cedcc9d302a in term 1.
I20240503 08:12:54.241593 30121 raft_consensus.cc:686] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Becoming Leader. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Running, Role: LEADER
I20240503 08:12:54.241892 30121 consensus_queue.cc:237] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } }
I20240503 08:12:54.247222 28230 catalog_manager.cc:5484] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: term changed from 0 to 1, leader changed from <none> to 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130). New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:54.293223 30103 raft_consensus.cc:482] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:54.293435 30103 raft_consensus.cc:504] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.294098 30103 leader_election.cc:288] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397), 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:12:54.294544 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e31346107a453a81921b23b244e7ff" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:54.294620 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e31346107a453a81921b23b244e7ff" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
I20240503 08:12:54.294803 28504 raft_consensus.cc:2454] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:54.294836 28634 raft_consensus.cc:2454] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 0.
I20240503 08:12:54.295239 28308 leader_election.cc:302] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:54.295573 30103 raft_consensus.cc:2781] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:54.295692 30103 raft_consensus.cc:482] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:54.295797 30103 raft_consensus.cc:3037] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.297796 30103 raft_consensus.cc:504] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.298916 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e31346107a453a81921b23b244e7ff" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:12:54.298911 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e31346107a453a81921b23b244e7ff" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:54.299134 28504 raft_consensus.cc:3037] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.299142 28634 raft_consensus.cc:3037] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.301985 28504 raft_consensus.cc:2454] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:54.301977 28634 raft_consensus.cc:2454] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 1.
I20240503 08:12:54.298731 30103 leader_election.cc:288] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Requested vote from peers 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397), 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:12:54.302482 28308 leader_election.cc:302] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:12:54.306155 30103 raft_consensus.cc:2781] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:54.306690 30103 raft_consensus.cc:686] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Becoming Leader. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Running, Role: LEADER
I20240503 08:12:54.307344 30103 consensus_queue.cc:237] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.309041 29986 raft_consensus.cc:482] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:54.309284 29986 raft_consensus.cc:504] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.310328 29986 leader_election.cc:288] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397), 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:12:54.311266 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9537f30aaf45adbc2cc4a784009de7" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:12:54.311650 28504 raft_consensus.cc:2454] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 0.
I20240503 08:12:54.313439 28699 leader_election.cc:302] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716ffbb2a00b4f43b03e5cedcc9d302a, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:54.313861 29986 raft_consensus.cc:2781] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:54.314020 29986 raft_consensus.cc:482] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:54.314121 29986 raft_consensus.cc:3037] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.314019 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9537f30aaf45adbc2cc4a784009de7" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d" is_pre_election: true
I20240503 08:12:54.314304 28634 raft_consensus.cc:2454] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 0.
I20240503 08:12:54.317683 29986 raft_consensus.cc:504] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.318578 29986 leader_election.cc:288] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 election: Requested vote from peers 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397), 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:12:54.319099 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9537f30aaf45adbc2cc4a784009de7" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:12:54.319319 28504 raft_consensus.cc:3037] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.319505 28634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9537f30aaf45adbc2cc4a784009de7" candidate_uuid: "93f7e733cec9415eb422b0a7ba8621cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:12:54.319736 28634 raft_consensus.cc:3037] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.320169 28230 catalog_manager.cc:5484] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129). New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:54.321707 28504 raft_consensus.cc:2454] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 1.
I20240503 08:12:54.322212 28699 leader_election.cc:302] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716ffbb2a00b4f43b03e5cedcc9d302a, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:12:54.322418 28634 raft_consensus.cc:2454] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7e733cec9415eb422b0a7ba8621cb in term 1.
I20240503 08:12:54.322554 29986 raft_consensus.cc:2781] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:54.323004 29986 raft_consensus.cc:686] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Becoming Leader. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Running, Role: LEADER
I20240503 08:12:54.323370 29986 consensus_queue.cc:237] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }
I20240503 08:12:54.326404 28230 catalog_manager.cc:5484] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132). New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:54.666265 30121 consensus_queue.cc:1035] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.673247 30121 consensus_queue.cc:1035] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.722160 30103 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:54.728789 30141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:54.729367 30141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:12:54.729293 30135 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:54.739404 30141 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 08:12:54.772468 30139 consensus_queue.cc:1035] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.780169 29986 consensus_queue.cc:1035] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 237.31 ms
time per row: 0.118655 ms
I20240503 08:12:55.156694 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.133:35121
--local_ip_for_outbound_sockets=127.20.107.133
--tserver_master_addrs=127.20.107.190:33183
--webserver_port=46727
--webserver_interface=127.20.107.133
--builtin_ntp_servers=127.20.107.148:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:12:55.545054 30199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:55.545500 30199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:55.545657 30199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:55.553650 30199 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:55.553802 30199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:55.553965 30199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.133
I20240503 08:12:55.563966 30199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.133:35121
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.107.133
--webserver_port=46727
--tserver_master_addrs=127.20.107.190:33183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.133
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:55.565573 30199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:55.567703 30199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:55.581848 30207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:55.582506 30205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:55.583177 30204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:55.584024 30199 server_base.cc:1034] running on GCE node
I20240503 08:12:55.584779 30199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:55.585813 30199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:55.587015 30199 hybrid_clock.cc:648] HybridClock initialized: now 1714723975586972 us; error 32 us; skew 500 ppm
I20240503 08:12:55.587335 30199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:55.589716 30199 webserver.cc:460] Webserver started at http://127.20.107.133:46727/ using document root <none> and password file <none>
I20240503 08:12:55.590595 30199 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:55.590811 30199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:55.596457 30199 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:12:55.599558 30213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:55.601166 30199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:55.601358 30199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c8375219addc48fa99ac474ed1c9aef9"
format_stamp: "Formatted at 2024-05-03 08:12:36 on dist-test-slave-4tl1"
I20240503 08:12:55.602208 30199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:55.628418 30199 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:55.629441 30199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:55.629786 30199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:55.630918 30199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:55.632756 30199 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:55.632846 30199 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:55.633011 30199 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:55.633069 30199 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:55.670339 30199 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.133:35121
I20240503 08:12:55.670394 30325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.133:35121 every 8 connection(s)
I20240503 08:12:55.671852 30199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:12:55.673285 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 30199
I20240503 08:12:55.674016 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.134:34587
--local_ip_for_outbound_sockets=127.20.107.134
--tserver_master_addrs=127.20.107.190:33183
--webserver_port=34649
--webserver_interface=127.20.107.134
--builtin_ntp_servers=127.20.107.148:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:12:55.686393 30326 heartbeater.cc:347] Connected to a master server at 127.20.107.190:33183
I20240503 08:12:55.686690 30326 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:55.687551 30326 heartbeater.cc:511] Master 127.20.107.190:33183 requested a full tablet report, sending...
I20240503 08:12:55.689198 28230 ts_manager.cc:194] Re-registered known tserver with Master: c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121)
I20240503 08:12:55.690439 28230 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.133:33911
W20240503 08:12:56.038327 30330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:56.038627 30330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:56.038717 30330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:56.046346 30330 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:56.046479 30330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:56.046607 30330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.134
I20240503 08:12:56.057096 30330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.134:34587
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.107.134
--webserver_port=34649
--tserver_master_addrs=127.20.107.190:33183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.134
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:56.058625 30330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:56.060732 30330 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:56.074986 30337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:56.075524 30336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:56.076217 30339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:56.077064 30330 server_base.cc:1034] running on GCE node
I20240503 08:12:56.077739 30330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:56.078758 30330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:56.079981 30330 hybrid_clock.cc:648] HybridClock initialized: now 1714723976079918 us; error 60 us; skew 500 ppm
I20240503 08:12:56.080335 30330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:56.082813 30330 webserver.cc:460] Webserver started at http://127.20.107.134:34649/ using document root <none> and password file <none>
I20240503 08:12:56.083560 30330 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:56.083693 30330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:56.089184 30330 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20240503 08:12:56.092306 30345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:56.093775 30330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:12:56.093935 30330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "21d21a5b596b4d48a17a1fb54a972844"
format_stamp: "Formatted at 2024-05-03 08:12:37 on dist-test-slave-4tl1"
I20240503 08:12:56.094822 30330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:56.133868 30330 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:56.134770 30330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:56.135049 30330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:56.136087 30330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:56.137686 30330 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:56.137760 30330 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:56.137884 30330 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:56.137934 30330 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:56.178284 30330 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.134:34587
I20240503 08:12:56.178344 30458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.134:34587 every 8 connection(s)
I20240503 08:12:56.179749 30330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:12:56.188373 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 30330
I20240503 08:12:56.188987 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.135:45937
--local_ip_for_outbound_sockets=127.20.107.135
--tserver_master_addrs=127.20.107.190:33183
--webserver_port=38853
--webserver_interface=127.20.107.135
--builtin_ntp_servers=127.20.107.148:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:12:56.194037 30459 heartbeater.cc:347] Connected to a master server at 127.20.107.190:33183
I20240503 08:12:56.194386 30459 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:56.195399 30459 heartbeater.cc:511] Master 127.20.107.190:33183 requested a full tablet report, sending...
I20240503 08:12:56.197135 28230 ts_manager.cc:194] Re-registered known tserver with Master: 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
I20240503 08:12:56.197925 28230 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.134:36577
W20240503 08:12:56.557809 30462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:56.558135 30462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:56.558290 30462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:56.566926 30462 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:56.567093 30462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:56.567278 30462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.135
I20240503 08:12:56.576901 30462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.135:45937
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.107.135
--webserver_port=38853
--tserver_master_addrs=127.20.107.190:33183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.135
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:56.578320 30462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:56.580190 30462 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:56.592480 30470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:56.593024 30467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:56.593825 30468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:56.594709 30462 server_base.cc:1034] running on GCE node
I20240503 08:12:56.595388 30462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:56.596443 30462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:56.597657 30462 hybrid_clock.cc:648] HybridClock initialized: now 1714723976597609 us; error 42 us; skew 500 ppm
I20240503 08:12:56.598026 30462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:56.600500 30462 webserver.cc:460] Webserver started at http://127.20.107.135:38853/ using document root <none> and password file <none>
I20240503 08:12:56.601287 30462 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:56.601411 30462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:56.606995 30462 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:12:56.609889 30476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:56.611559 30462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:56.611754 30462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "bea2145b8e24436392b18ec3cfe90b93"
format_stamp: "Formatted at 2024-05-03 08:12:37 on dist-test-slave-4tl1"
I20240503 08:12:56.612545 30462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:56.642727 30462 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:56.643496 30462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:56.643750 30462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:56.644610 30462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:56.646049 30462 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:56.646121 30462 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:56.646215 30462 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:56.646258 30462 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:56.677783 30462 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.135:45937
I20240503 08:12:56.677848 30588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.135:45937 every 8 connection(s)
I20240503 08:12:56.679189 30462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:12:56.685788 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 30462
I20240503 08:12:56.686544 20910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
/tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.107.136:42389
--local_ip_for_outbound_sockets=127.20.107.136
--tserver_master_addrs=127.20.107.190:33183
--webserver_port=37883
--webserver_interface=127.20.107.136
--builtin_ntp_servers=127.20.107.148:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:12:56.691673 30589 heartbeater.cc:347] Connected to a master server at 127.20.107.190:33183
I20240503 08:12:56.691977 30589 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:56.692754 30589 heartbeater.cc:511] Master 127.20.107.190:33183 requested a full tablet report, sending...
I20240503 08:12:56.693130 30326 heartbeater.cc:503] Master 127.20.107.190:33183 was elected leader, sending a full tablet report...
I20240503 08:12:56.694186 28233 ts_manager.cc:194] Re-registered known tserver with Master: bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937)
I20240503 08:12:56.695065 28233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.135:48101
W20240503 08:12:57.048156 30592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:57.048462 30592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:57.048591 30592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:57.056681 30592 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:57.056818 30592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:57.056954 30592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.107.136
I20240503 08:12:57.067528 30592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.107.148:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.107.136:42389
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.107.136
--webserver_port=37883
--tserver_master_addrs=127.20.107.190:33183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.107.136
--log_dir=/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:57.068846 30592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:57.070792 30592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:57.084107 30599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:57.084628 30600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:57.085256 30602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:57.086251 30592 server_base.cc:1034] running on GCE node
I20240503 08:12:57.086905 30592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:57.087883 30592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:57.089082 30592 hybrid_clock.cc:648] HybridClock initialized: now 1714723977089036 us; error 40 us; skew 500 ppm
I20240503 08:12:57.089390 30592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:57.091782 30592 webserver.cc:460] Webserver started at http://127.20.107.136:37883/ using document root <none> and password file <none>
I20240503 08:12:57.092509 30592 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:57.092633 30592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:57.098402 30592 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 08:12:57.101346 30608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:57.102628 30592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:12:57.102775 30592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "b29361eb4afa4ab3af7d8019d8a5042d"
format_stamp: "Formatted at 2024-05-03 08:12:38 on dist-test-slave-4tl1"
I20240503 08:12:57.103554 30592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:57.130242 30592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:57.131035 30592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:57.131282 30592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:57.132186 30592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:57.133780 30592 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:57.133883 30592 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:57.134014 30592 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:57.134074 30592 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:57.166257 30592 rpc_server.cc:307] RPC server started. Bound to: 127.20.107.136:42389
I20240503 08:12:57.166287 30720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.107.136:42389 every 8 connection(s)
I20240503 08:12:57.168061 30592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:12:57.168565 20910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu as pid 30592
I20240503 08:12:57.201131 30459 heartbeater.cc:503] Master 127.20.107.190:33183 was elected leader, sending a full tablet report...
I20240503 08:12:57.230463 30721 heartbeater.cc:347] Connected to a master server at 127.20.107.190:33183
I20240503 08:12:57.230813 30721 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:57.231889 30721 heartbeater.cc:511] Master 127.20.107.190:33183 requested a full tablet report, sending...
I20240503 08:12:57.233821 28233 ts_manager.cc:194] Re-registered known tserver with Master: b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389)
I20240503 08:12:57.234870 28233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.107.136:47481
I20240503 08:12:57.697448 30589 heartbeater.cc:503] Master 127.20.107.190:33183 was elected leader, sending a full tablet report...
I20240503 08:12:58.238070 30721 heartbeater.cc:503] Master 127.20.107.190:33183 was elected leader, sending a full tablet report...
I20240503 08:12:58.475534 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:12:58.488912 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:58.500470 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:12:58.568327 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:12:58.602301 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:58.625326 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:58.627663 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:12:58.642550 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:58.667160 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:12:58.698908 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:12:58.774746 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:58.824301 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:58.826424 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:58.854575 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:58.874867 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:58.918462 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:59.083429 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:59.109601 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:12:59.150169 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:12:59.213999 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:59.251760 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:59.300628 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:12:59.350550 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:59.359943 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:59.423374 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:12:59.453688 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:59.532413 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:59.535027 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:59.549213 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:59.561492 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:12:59.561851 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:12:59.562341 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:12:59.571574 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:59.571754 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:12:59.585248 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:59.595240 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:59.643656 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:12:59.644003 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:59.646801 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:59.657734 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:00.296622 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:00.337246 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:00.361063 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:00.362721 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:00.375550 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:00.408613 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:00.468492 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:00.468605 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:00.481948 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:00.519502 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:00.539192 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:00.550231 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:00.570036 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:00.614955 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:00.618391 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:00.677829 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:01.116758 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:01.135660 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:01.153260 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:01.240394 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:01.357038 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:01.357578 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:01.362427 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:01.404093 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:01.411161 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:01.423437 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:01.427277 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:01.458268 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:01.533768 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:01.534422 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:01.568464 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:01.584405 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:01.598448 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:01.620093 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:01.637993 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:01.670755 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:01.683485 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:01.719285 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:01.765261 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:01.803755 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:02.374859 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:02.414541 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:02.419631 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:02.449468 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:02.472561 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:02.483301 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:02.509289 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:02.550211 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:02.583143 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:02.587590 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:02.605123 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:02.605664 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:02.636415 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:02.658411 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:02.664252 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:02.676082 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:02.775684 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:02.792578 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:02.822746 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:02.889350 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:02.896721 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:02.906368 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:02.927035 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:03.003931 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:03.081350 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:03.122579 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:03.130249 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:03.145900 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:03.153239 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:03.165599 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:03.173708 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:03.185261 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:03.204838 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:03.225160 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:03.235221 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:03.237031 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:03.241046 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:03.268595 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:03.276579 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:03.335373 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:03.536571 28373 consensus_queue.cc:237] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:03.553014 28634 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:03.554296 28764 consensus_queue.cc:237] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:03.557237 31011 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:03.573253 31484 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:03.574141 28634 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:03.580762 28503 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:03.581660 28228 catalog_manager.cc:5484] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:03.587270 31011 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:03.611589 28503 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:03.613301 30995 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:03.629192 30995 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:03.636152 28504 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:03.646303 28374 consensus_queue.cc:237] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
W20240503 08:13:03.650624 28309 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2ebb0b8475b4ed9a09711a87867de08. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.657128 28634 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:03.665030 28504 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:03.665696 28765 consensus_queue.cc:237] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:03.667747 31486 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:03.669000 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48e31346107a453a81921b23b244e7ff. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.671052 28634 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:03.671470 31465 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:03.674724 28504 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:03.675580 31464 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:03.677558 31486 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:03.688673 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6e1b9f104ea471cab7527bd56433327. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.692184 28635 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:03.696333 31464 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:03.698115 31465 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:03.698726 31488 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:03.707235 28635 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:03.710927 28633 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:03.710808 28502 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:03.717873 28505 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:03.718955 28634 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:03.726001 28505 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
W20240503 08:13:03.731279 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0caf86850b3841ea811b0a6ab19ae330. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.737833 28232 catalog_manager.cc:5484] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:03.752717 28230 catalog_manager.cc:5484] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:03.763654 28232 catalog_manager.cc:5484] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:03.795213 28374 consensus_queue.cc:237] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:03.798712 28765 consensus_queue.cc:237] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
W20240503 08:13:03.802361 28309 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2ebb0b8475b4ed9a09711a87867de08. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.805321 28505 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:03.806352 31488 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:03.810616 31486 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:03.811576 28634 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:03.812348 31011 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:03.814325 28634 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:03.814718 28505 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:03.821357 28232 catalog_manager.cc:5484] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 4 to 5, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
W20240503 08:13:03.822871 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2ebb0b8475b4ed9a09711a87867de08. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.850018 28634 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:03.850775 31464 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:03.861111 28373 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:03.862708 31464 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:03.865142 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bfa8537bc8a4950b53fbf3c36d5ecf3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.871510 28373 consensus_queue.cc:237] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:03.875645 31504 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:03.876430 28374 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
W20240503 08:13:03.879042 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2ebb0b8475b4ed9a09711a87867de08. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.879102 28505 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 08:13:03.879896 28309 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2ebb0b8475b4ed9a09711a87867de08. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.880200 31488 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:03.883548 28232 catalog_manager.cc:5484] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:13:03.891602 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2ebb0b8475b4ed9a09711a87867de08. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.892081 31490 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:03.892849 28505 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:03.895813 28633 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:03.900306 31488 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:03.900195 28232 catalog_manager.cc:5484] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 5 to 6, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:03.904089 28633 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:03.908288 28635 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:03.933465 28765 consensus_queue.cc:237] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:03.935834 28634 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:13:03.936381 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bfa8537bc8a4950b53fbf3c36d5ecf3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.936743 30995 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:03.936887 28373 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:13:03.937474 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bfa8537bc8a4950b53fbf3c36d5ecf3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.937786 30995 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:03.940253 31464 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:03.946669 28635 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:03.954742 28228 catalog_manager.cc:5484] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:03.954515 28374 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:03.992455 28762 consensus_queue.cc:237] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
W20240503 08:13:03.996275 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e9537f30aaf45adbc2cc4a784009de7. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.996910 28505 raft_consensus.cc:1261] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:03.997650 31504 consensus_queue.cc:1035] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.025503 31504 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:04.029578 28505 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:04.036568 28232 catalog_manager.cc:5484] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:04.043438 28635 raft_consensus.cc:1261] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.045017 31504 consensus_queue.cc:1035] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.054159 28505 consensus_queue.cc:237] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:04.059403 28635 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:04.074383 28762 raft_consensus.cc:1261] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.075251 28635 raft_consensus.cc:1261] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.075825 31539 consensus_queue.cc:1035] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.105530 31597 ts_tablet_manager.cc:902] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:13:04.105716 31539 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:04.111552 28634 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:04.121348 31598 ts_tablet_manager.cc:902] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:13:04.126834 28228 catalog_manager.cc:5484] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:04.162681 31380 consensus_queue.cc:1035] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.163635 31601 ts_tablet_manager.cc:902] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:04.171345 28501 consensus_queue.cc:237] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:04.172116 28765 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:04.193259 31598 tablet_copy_client.cc:308] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:13:04.214087 28374 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.214810 31380 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.225759 28765 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.226691 31380 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.237766 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet c6e1b9f104ea471cab7527bd56433327 from peer c8375219addc48fa99ac474ed1c9aef9 ({username='slave'} at 127.20.107.133:41339)
I20240503 08:13:04.238157 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet c6e1b9f104ea471cab7527bd56433327 from peer c8375219addc48fa99ac474ed1c9aef9 at {username='slave'} at 127.20.107.133:41339: session id = c8375219addc48fa99ac474ed1c9aef9-c6e1b9f104ea471cab7527bd56433327
I20240503 08:13:04.242792 28785 tablet_copy_source_session.cc:205] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:04.253800 28373 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:04.262593 28232 catalog_manager.cc:5484] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:04.252736 31380 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:04.288584 31598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e1b9f104ea471cab7527bd56433327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.306030 28762 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:04.336733 31597 tablet_copy_client.cc:308] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
W20240503 08:13:04.368238 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 320d1e880958427a88e93321171cb36e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:04.371078 28374 consensus_queue.cc:237] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
W20240503 08:13:04.375805 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48e31346107a453a81921b23b244e7ff. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:04.376506 28634 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:04.377318 31490 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:04.380638 28634 consensus_queue.cc:237] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:04.382803 31490 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:04.385016 28635 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:04.410140 28229 catalog_manager.cc:5484] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:04.352676 31650 ts_tablet_manager.cc:902] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:04.356212 31646 ts_tablet_manager.cc:902] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:13:04.433413 31646 tablet_copy_client.cc:308] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
W20240503 08:13:04.453193 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48e31346107a453a81921b23b244e7ff. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:04.455243 28505 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:04.458032 31486 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:04.467489 31657 ts_tablet_manager.cc:902] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:04.472270 28374 raft_consensus.cc:1261] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.473273 31579 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.485049 28762 raft_consensus.cc:1261] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.485817 31579 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.496120 31579 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:04.497431 31659 ts_tablet_manager.cc:902] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:13:04.499512 31670 ts_tablet_manager.cc:902] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:13:04.500829 28373 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:04.507620 28228 catalog_manager.cc:5484] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:04.508980 28505 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:04.507174 31650 tablet_copy_client.cc:308] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:04.528379 28762 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
W20240503 08:13:04.537825 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 585ba864cc384e7f9db90e9d4c47e82c. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:04.540371 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a20489bf65774902af38d228ee966ffe. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:04.543572 31672 ts_tablet_manager.cc:902] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:04.544723 31692 ts_tablet_manager.cc:902] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:13:04.554370 31657 tablet_copy_client.cc:308] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:04.563298 31598 tablet_copy_client.cc:783] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:04.563879 31598 tablet_copy_client.cc:647] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:04.569120 31598 tablet_copy_client.cc:523] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:04.569638 31670 tablet_copy_client.cc:308] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:13:04.576164 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 0caf86850b3841ea811b0a6ab19ae330 from peer 21d21a5b596b4d48a17a1fb54a972844 ({username='slave'} at 127.20.107.134:58893)
I20240503 08:13:04.576401 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 0caf86850b3841ea811b0a6ab19ae330 from peer 21d21a5b596b4d48a17a1fb54a972844 at {username='slave'} at 127.20.107.134:58893: session id = 21d21a5b596b4d48a17a1fb54a972844-0caf86850b3841ea811b0a6ab19ae330
I20240503 08:13:04.580516 28785 tablet_copy_source_session.cc:205] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:04.582686 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3 from peer 21d21a5b596b4d48a17a1fb54a972844 ({username='slave'} at 127.20.107.134:58893)
I20240503 08:13:04.582940 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3 from peer 21d21a5b596b4d48a17a1fb54a972844 at {username='slave'} at 127.20.107.134:58893: session id = 21d21a5b596b4d48a17a1fb54a972844-3bfa8537bc8a4950b53fbf3c36d5ecf3
I20240503 08:13:04.586833 28785 tablet_copy_source_session.cc:205] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:04.593622 31597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caf86850b3841ea811b0a6ab19ae330. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.594543 31692 tablet_copy_client.cc:308] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:13:04.612498 31598 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap starting.
I20240503 08:13:04.615839 31659 tablet_copy_client.cc:308] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:13:04.619172 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 6e9537f30aaf45adbc2cc4a784009de7 from peer c8375219addc48fa99ac474ed1c9aef9 ({username='slave'} at 127.20.107.133:41339)
I20240503 08:13:04.619374 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 6e9537f30aaf45adbc2cc4a784009de7 from peer c8375219addc48fa99ac474ed1c9aef9 at {username='slave'} at 127.20.107.133:41339: session id = c8375219addc48fa99ac474ed1c9aef9-6e9537f30aaf45adbc2cc4a784009de7
I20240503 08:13:04.622896 28785 tablet_copy_source_session.cc:205] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:04.625615 31659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9537f30aaf45adbc2cc4a784009de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.617071 31672 tablet_copy_client.cc:308] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:04.633639 28784 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3 from peer b29361eb4afa4ab3af7d8019d8a5042d ({username='slave'} at 127.20.107.136:41343)
I20240503 08:13:04.633864 28784 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3 from peer b29361eb4afa4ab3af7d8019d8a5042d at {username='slave'} at 127.20.107.136:41343: session id = b29361eb4afa4ab3af7d8019d8a5042d-3bfa8537bc8a4950b53fbf3c36d5ecf3
I20240503 08:13:04.635818 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.636787 28784 tablet_copy_source_session.cc:205] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:04.637341 28525 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet 320d1e880958427a88e93321171cb36e from peer b29361eb4afa4ab3af7d8019d8a5042d ({username='slave'} at 127.20.107.136:33075)
I20240503 08:13:04.637852 28525 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet 320d1e880958427a88e93321171cb36e from peer b29361eb4afa4ab3af7d8019d8a5042d at {username='slave'} at 127.20.107.136:33075: session id = b29361eb4afa4ab3af7d8019d8a5042d-320d1e880958427a88e93321171cb36e
I20240503 08:13:04.644323 28525 tablet_copy_source_session.cc:205] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:04.657024 31670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320d1e880958427a88e93321171cb36e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.670526 31692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.670385 28505 consensus_queue.cc:237] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:04.685704 31726 ts_tablet_manager.cc:902] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:13:04.702307 28765 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:04.703228 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0d07a6278d545e69bf7d2012109d1f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:04.703573 31380 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.706483 31646 tablet_copy_client.cc:783] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:04.707077 31646 tablet_copy_client.cc:647] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:04.710176 31601 tablet_copy_client.cc:308] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:04.714869 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet b2ebb0b8475b4ed9a09711a87867de08 from peer c8375219addc48fa99ac474ed1c9aef9 ({username='slave'} at 127.20.107.133:49403)
I20240503 08:13:04.715219 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet b2ebb0b8475b4ed9a09711a87867de08 from peer c8375219addc48fa99ac474ed1c9aef9 at {username='slave'} at 127.20.107.133:49403: session id = c8375219addc48fa99ac474ed1c9aef9-b2ebb0b8475b4ed9a09711a87867de08
I20240503 08:13:04.719543 28394 tablet_copy_source_session.cc:205] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:04.723004 31672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ebb0b8475b4ed9a09711a87867de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.728778 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet b2ebb0b8475b4ed9a09711a87867de08 from peer 21d21a5b596b4d48a17a1fb54a972844 ({username='slave'} at 127.20.107.134:32851)
I20240503 08:13:04.729148 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet b2ebb0b8475b4ed9a09711a87867de08 from peer 21d21a5b596b4d48a17a1fb54a972844 at {username='slave'} at 127.20.107.134:32851: session id = 21d21a5b596b4d48a17a1fb54a972844-b2ebb0b8475b4ed9a09711a87867de08
I20240503 08:13:04.731899 31659 tablet_copy_client.cc:783] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:04.732204 31659 tablet_copy_client.cc:647] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:04.736222 31659 tablet_copy_client.cc:523] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:04.736817 28394 tablet_copy_source_session.cc:205] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:04.740339 31646 tablet_copy_client.cc:523] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:04.746471 31597 tablet_copy_client.cc:783] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:04.746767 31597 tablet_copy_client.cc:647] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:04.752174 31646 tablet_bootstrap.cc:492] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap starting.
I20240503 08:13:04.751930 28374 consensus_queue.cc:237] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:04.753365 28373 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.758404 31726 tablet_copy_client.cc:308] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:13:04.760387 31539 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.761071 31659 tablet_bootstrap.cc:492] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap starting.
I20240503 08:13:04.764663 31594 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:04.765542 28762 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:04.773245 28229 catalog_manager.cc:5484] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:04.774961 28393 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet 48e31346107a453a81921b23b244e7ff from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:40941)
I20240503 08:13:04.775215 28393 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet 48e31346107a453a81921b23b244e7ff from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:40941: session id = bea2145b8e24436392b18ec3cfe90b93-48e31346107a453a81921b23b244e7ff
I20240503 08:13:04.778344 28393 tablet_copy_source_session.cc:205] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 08:13:04.781587 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ede24a27608a4cf196dd46f85c4163e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:04.782583 28505 raft_consensus.cc:1261] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.783360 31490 consensus_queue.cc:1035] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.784344 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet b2ebb0b8475b4ed9a09711a87867de08 from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:40941)
I20240503 08:13:04.784538 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet b2ebb0b8475b4ed9a09711a87867de08 from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:40941: session id = bea2145b8e24436392b18ec3cfe90b93-b2ebb0b8475b4ed9a09711a87867de08
I20240503 08:13:04.802493 28525 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet a20489bf65774902af38d228ee966ffe from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:59971)
I20240503 08:13:04.802767 28525 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet a20489bf65774902af38d228ee966ffe from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:59971: session id = bea2145b8e24436392b18ec3cfe90b93-a20489bf65774902af38d228ee966ffe
I20240503 08:13:04.803337 28762 raft_consensus.cc:1261] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.803871 31670 tablet_copy_client.cc:783] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:04.804050 31490 consensus_queue.cc:1035] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.805220 31670 tablet_copy_client.cc:647] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:04.806032 28525 tablet_copy_source_session.cc:205] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:04.813064 31670 tablet_copy_client.cc:523] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:04.824697 31486 raft_consensus.cc:2932] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:04.834393 31601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e31346107a453a81921b23b244e7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.835036 28229 catalog_manager.cc:5484] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:04.837468 28765 raft_consensus.cc:2932] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:04.850457 31670 tablet_bootstrap.cc:492] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap starting.
I20240503 08:13:04.754191 31650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ebb0b8475b4ed9a09711a87867de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.866605 31753 ts_tablet_manager.cc:902] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:04.873090 31726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20489bf65774902af38d228ee966ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.875056 31650 tablet_copy_client.cc:783] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:04.875334 31650 tablet_copy_client.cc:647] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:04.878909 31650 tablet_copy_client.cc:523] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:04.879940 31597 tablet_copy_client.cc:523] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:04.882725 28505 raft_consensus.cc:2932] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:04.888835 31598 log.cc:824] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:04.893539 28372 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:04.895872 31650 tablet_bootstrap.cc:492] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap starting.
I20240503 08:13:04.896281 31597 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap starting.
I20240503 08:13:04.897406 28394 tablet_copy_source_session.cc:205] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:04.901197 31657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ebb0b8475b4ed9a09711a87867de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.918442 31771 ts_tablet_manager.cc:902] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:13:04.927109 31672 tablet_copy_client.cc:783] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:04.927605 31672 tablet_copy_client.cc:647] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:04.928982 31753 tablet_copy_client.cc:308] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:04.929718 28393 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet 48e31346107a453a81921b23b244e7ff from peer 21d21a5b596b4d48a17a1fb54a972844 ({username='slave'} at 127.20.107.134:32851)
I20240503 08:13:04.929926 28393 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet 48e31346107a453a81921b23b244e7ff from peer 21d21a5b596b4d48a17a1fb54a972844 at {username='slave'} at 127.20.107.134:32851: session id = 21d21a5b596b4d48a17a1fb54a972844-48e31346107a453a81921b23b244e7ff
I20240503 08:13:04.932354 31692 tablet_copy_client.cc:783] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:04.933208 28393 tablet_copy_source_session.cc:205] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:04.935145 31672 tablet_copy_client.cc:523] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:04.935163 31657 tablet_copy_client.cc:783] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:04.935623 31753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e31346107a453a81921b23b244e7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.935849 31692 tablet_copy_client.cc:647] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:04.937065 31657 tablet_copy_client.cc:647] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:04.938210 31601 tablet_copy_client.cc:783] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:04.938483 31601 tablet_copy_client.cc:647] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:04.940682 31726 tablet_copy_client.cc:783] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:04.941589 31692 tablet_copy_client.cc:523] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:04.941787 31657 tablet_copy_client.cc:523] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:04.942878 31726 tablet_copy_client.cc:647] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:04.945467 31692 tablet_bootstrap.cc:492] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap starting.
I20240503 08:13:04.946400 31726 tablet_copy_client.cc:523] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:04.950676 31657 tablet_bootstrap.cc:492] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
I20240503 08:13:04.964195 31672 tablet_bootstrap.cc:492] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap starting.
I20240503 08:13:04.968756 31601 tablet_copy_client.cc:523] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:04.971807 28765 consensus_queue.cc:237] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:05.011693 28505 consensus_queue.cc:237] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:05.017108 31601 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
I20240503 08:13:05.018383 31726 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
W20240503 08:13:05.030730 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:05.036144 28634 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:05.037822 31464 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:05.059293 28501 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:05.059999 31465 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:05.066458 31465 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:05.085151 31646 log.cc:824] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:05.085561 31670 log.cc:824] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:05.085805 28635 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:05.096482 31753 tablet_copy_client.cc:783] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:05.099776 31771 tablet_copy_client.cc:308] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:13:05.109014 31753 tablet_copy_client.cc:647] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:05.111263 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet 585ba864cc384e7f9db90e9d4c47e82c from peer c8375219addc48fa99ac474ed1c9aef9 ({username='slave'} at 127.20.107.133:44309)
I20240503 08:13:05.111649 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet 585ba864cc384e7f9db90e9d4c47e82c from peer c8375219addc48fa99ac474ed1c9aef9 at {username='slave'} at 127.20.107.133:44309: session id = c8375219addc48fa99ac474ed1c9aef9-585ba864cc384e7f9db90e9d4c47e82c
I20240503 08:13:05.116086 28655 tablet_copy_source_session.cc:205] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:05.118932 31771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585ba864cc384e7f9db90e9d4c47e82c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.134074 28502 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:05.137418 31753 tablet_copy_client.cc:523] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:05.141703 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:05.140692 28232 catalog_manager.cc:5484] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 4 to 5, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
W20240503 08:13:05.151795 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac392ff7965a405f96efc777aca4f0bb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:05.152890 31753 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap starting.
I20240503 08:13:05.154556 28372 raft_consensus.cc:1261] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:05.155133 31539 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:05.175184 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:05.180788 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:05.181627 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:05.205693 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:05.211475 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:05.236057 31828 ts_tablet_manager.cc:902] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:13:05.239933 28765 raft_consensus.cc:1261] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:05.242437 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:05.245195 31380 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:05.253279 31594 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:05.257445 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:05.268576 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:05.277163 31771 tablet_copy_client.cc:783] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:05.279381 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:05.283789 28374 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:05.289141 31771 tablet_copy_client.cc:647] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:05.296751 31771 tablet_copy_client.cc:523] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:05.299135 31601 log.cc:824] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:05.305107 31771 tablet_bootstrap.cc:492] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9: Bootstrap starting.
I20240503 08:13:05.334396 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:13:05.339670 28700 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6e1b9f104ea471cab7527bd56433327. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:05.346483 28232 catalog_manager.cc:5484] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:05.375739 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:05.375126 28765 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:05.382988 31770 ts_tablet_manager.cc:902] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:05.384814 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:05.428438 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:05.431592 31828 tablet_copy_client.cc:308] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:13:05.432183 28761 consensus_queue.cc:237] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:05.432447 28525 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet b0d07a6278d545e69bf7d2012109d1f4 from peer b29361eb4afa4ab3af7d8019d8a5042d ({username='slave'} at 127.20.107.136:33075)
I20240503 08:13:05.432631 28525 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet b0d07a6278d545e69bf7d2012109d1f4 from peer b29361eb4afa4ab3af7d8019d8a5042d at {username='slave'} at 127.20.107.136:33075: session id = b29361eb4afa4ab3af7d8019d8a5042d-b0d07a6278d545e69bf7d2012109d1f4
I20240503 08:13:05.436501 28525 tablet_copy_source_session.cc:205] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:05.438016 31828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d07a6278d545e69bf7d2012109d1f4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:13:05.444793 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4babf0ea9b954a23b1b1c4e403dce0bd. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:05.445354 28635 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:05.445969 31465 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:05.448331 30524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:05.448503 31770 tablet_copy_client.cc:308] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:05.453943 28501 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:05.456480 31465 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:05.472987 31828 tablet_copy_client.cc:783] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:05.477440 31828 tablet_copy_client.cc:647] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:05.482452 31465 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:05.487365 31828 tablet_copy_client.cc:523] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:05.488147 28501 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:05.488705 28634 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:05.490706 28232 catalog_manager.cc:5484] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:05.493364 31828 tablet_bootstrap.cc:492] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap starting.
I20240503 08:13:05.513188 30523 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:05.537650 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet ede24a27608a4cf196dd46f85c4163e1 from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:40941)
I20240503 08:13:05.545157 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet ede24a27608a4cf196dd46f85c4163e1 from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:40941: session id = bea2145b8e24436392b18ec3cfe90b93-ede24a27608a4cf196dd46f85c4163e1
I20240503 08:13:05.554033 28394 tablet_copy_source_session.cc:205] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:05.556003 31770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede24a27608a4cf196dd46f85c4163e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.588820 31770 tablet_copy_client.cc:783] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:05.590893 31770 tablet_copy_client.cc:647] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:05.640314 31672 tablet_bootstrap.cc:492] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:05.662330 31770 tablet_copy_client.cc:523] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:05.666083 31672 tablet_bootstrap.cc:492] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap complete.
I20240503 08:13:05.666718 31672 ts_tablet_manager.cc:1367] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9: Time spent bootstrapping tablet: real 0.703s user 0.061s sys 0.000s
I20240503 08:13:05.670104 31598 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:05.670604 31598 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap complete.
I20240503 08:13:05.670902 31598 ts_tablet_manager.cc:1367] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9: Time spent bootstrapping tablet: real 1.060s user 0.057s sys 0.013s
I20240503 08:13:05.681183 28501 consensus_queue.cc:237] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:05.695219 31770 tablet_bootstrap.cc:492] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
I20240503 08:13:05.698781 31672 raft_consensus.cc:348] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:05.700951 31672 raft_consensus.cc:729] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8375219addc48fa99ac474ed1c9aef9, State: Initialized, Role: LEARNER
I20240503 08:13:05.701648 31672 consensus_queue.cc:260] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:05.702688 31877 ts_tablet_manager.cc:902] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:13:05.705793 31672 ts_tablet_manager.cc:1398] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9: Time spent starting tablet: real 0.039s user 0.000s sys 0.007s
I20240503 08:13:05.706595 31659 tablet_bootstrap.cc:492] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:05.708848 31659 tablet_bootstrap.cc:492] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap complete.
I20240503 08:13:05.709837 31659 ts_tablet_manager.cc:1367] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9: Time spent bootstrapping tablet: real 0.953s user 0.066s sys 0.006s
I20240503 08:13:05.712080 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session c8375219addc48fa99ac474ed1c9aef9-b2ebb0b8475b4ed9a09711a87867de08 received from {username='slave'} at 127.20.107.133:49403
I20240503 08:13:05.712245 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session c8375219addc48fa99ac474ed1c9aef9-b2ebb0b8475b4ed9a09711a87867de08 on tablet b2ebb0b8475b4ed9a09711a87867de08 with peer c8375219addc48fa99ac474ed1c9aef9
I20240503 08:13:05.717109 31659 raft_consensus.cc:348] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:05.724567 31659 raft_consensus.cc:729] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8375219addc48fa99ac474ed1c9aef9, State: Initialized, Role: LEARNER
I20240503 08:13:05.716578 31598 raft_consensus.cc:348] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:05.725040 31598 raft_consensus.cc:729] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8375219addc48fa99ac474ed1c9aef9, State: Initialized, Role: LEARNER
I20240503 08:13:05.725248 31598 consensus_queue.cc:260] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:05.725775 31598 ts_tablet_manager.cc:1398] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9: Time spent starting tablet: real 0.055s user 0.008s sys 0.000s
I20240503 08:13:05.724859 31659 consensus_queue.cc:260] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:05.729297 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session c8375219addc48fa99ac474ed1c9aef9-c6e1b9f104ea471cab7527bd56433327 received from {username='slave'} at 127.20.107.133:41339
I20240503 08:13:05.729503 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session c8375219addc48fa99ac474ed1c9aef9-c6e1b9f104ea471cab7527bd56433327 on tablet c6e1b9f104ea471cab7527bd56433327 with peer c8375219addc48fa99ac474ed1c9aef9
I20240503 08:13:05.736194 31886 ts_tablet_manager.cc:902] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:13:05.738260 31659 ts_tablet_manager.cc:1398] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9: Time spent starting tablet: real 0.028s user 0.002s sys 0.002s
I20240503 08:13:05.739722 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session c8375219addc48fa99ac474ed1c9aef9-6e9537f30aaf45adbc2cc4a784009de7 received from {username='slave'} at 127.20.107.133:41339
I20240503 08:13:05.739879 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session c8375219addc48fa99ac474ed1c9aef9-6e9537f30aaf45adbc2cc4a784009de7 on tablet 6e9537f30aaf45adbc2cc4a784009de7 with peer c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:05.741678 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:05.742436 28374 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:05.743427 31833 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:05.746076 31594 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:05.750249 28762 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:05.751542 31539 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:05.753345 28763 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:05.760639 28374 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:05.770999 28228 catalog_manager.cc:5484] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 4 to 5, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:05.830483 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:05.862835 31886 tablet_copy_client.cc:308] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:13:05.864575 28525 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet ac392ff7965a405f96efc777aca4f0bb from peer b29361eb4afa4ab3af7d8019d8a5042d ({username='slave'} at 127.20.107.136:33075)
I20240503 08:13:05.864835 28525 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet ac392ff7965a405f96efc777aca4f0bb from peer b29361eb4afa4ab3af7d8019d8a5042d at {username='slave'} at 127.20.107.136:33075: session id = b29361eb4afa4ab3af7d8019d8a5042d-ac392ff7965a405f96efc777aca4f0bb
I20240503 08:13:05.870931 28525 tablet_copy_source_session.cc:205] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:05.874730 30280 raft_consensus.cc:1203] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:05.878917 31886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac392ff7965a405f96efc777aca4f0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.886875 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:05.899499 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:05.937830 31886 tablet_copy_client.cc:783] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:05.938364 31886 tablet_copy_client.cc:647] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:05.944078 31886 tablet_copy_client.cc:523] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:05.950027 31886 tablet_bootstrap.cc:492] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap starting.
I20240503 08:13:05.950830 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:05.953882 30280 raft_consensus.cc:1203] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 08:13:05.966468 31911 ts_tablet_manager.cc:902] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:13:05.970583 31670 tablet_bootstrap.cc:492] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:05.971258 31670 tablet_bootstrap.cc:492] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap complete.
I20240503 08:13:05.971972 31670 ts_tablet_manager.cc:1367] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d: Time spent bootstrapping tablet: real 1.122s user 0.054s sys 0.019s
I20240503 08:13:05.977509 28634 consensus_queue.cc:237] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:06.004827 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:06.014688 28374 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:06.035804 31695 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:06.038434 28501 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:06.039999 31365 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:06.047919 31650 tablet_bootstrap.cc:492] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:06.048523 31650 tablet_bootstrap.cc:492] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap complete.
W20240503 08:13:06.057008 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0d07a6278d545e69bf7d2012109d1f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:06.094882 31771 tablet_bootstrap.cc:492] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:06.095389 31771 tablet_bootstrap.cc:492] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9: Bootstrap complete.
I20240503 08:13:06.095697 31771 ts_tablet_manager.cc:1367] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9: Time spent bootstrapping tablet: real 0.791s user 0.058s sys 0.000s
I20240503 08:13:06.096556 31771 raft_consensus.cc:348] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:06.097532 31771 raft_consensus.cc:729] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8375219addc48fa99ac474ed1c9aef9, State: Initialized, Role: LEARNER
I20240503 08:13:06.097797 31771 consensus_queue.cc:260] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:06.098675 31771 ts_tablet_manager.cc:1398] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:06.099594 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session c8375219addc48fa99ac474ed1c9aef9-585ba864cc384e7f9db90e9d4c47e82c received from {username='slave'} at 127.20.107.133:44309
I20240503 08:13:06.099807 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session c8375219addc48fa99ac474ed1c9aef9-585ba864cc384e7f9db90e9d4c47e82c on tablet 585ba864cc384e7f9db90e9d4c47e82c with peer c8375219addc48fa99ac474ed1c9aef9
I20240503 08:13:06.105180 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:06.104570 30280 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:06.117944 31597 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:06.117807 31670 raft_consensus.cc:348] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:06.118508 31597 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap complete.
I20240503 08:13:06.118490 31670 raft_consensus.cc:729] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b29361eb4afa4ab3af7d8019d8a5042d, State: Initialized, Role: LEARNER
I20240503 08:13:06.118880 31597 ts_tablet_manager.cc:1367] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844: Time spent bootstrapping tablet: real 1.228s user 0.062s sys 0.000s
I20240503 08:13:06.119139 31670 consensus_queue.cc:260] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:06.121553 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:06.126214 31597 raft_consensus.cc:348] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:06.126869 31597 raft_consensus.cc:729] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Initialized, Role: LEARNER
I20240503 08:13:06.127475 31597 consensus_queue.cc:260] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:06.146277 31650 ts_tablet_manager.cc:1367] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844: Time spent bootstrapping tablet: real 1.251s user 0.043s sys 0.023s
I20240503 08:13:06.154568 31365 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:06.156361 31650 raft_consensus.cc:348] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:06.156996 31650 raft_consensus.cc:729] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Initialized, Role: LEARNER
I20240503 08:13:06.157450 31650 consensus_queue.cc:260] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:06.157820 28505 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:06.177371 30523 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:06.183612 28229 catalog_manager.cc:5484] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:06.204152 31670 ts_tablet_manager.cc:1398] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d: Time spent starting tablet: real 0.232s user 0.013s sys 0.000s
I20240503 08:13:06.205389 28525 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-320d1e880958427a88e93321171cb36e received from {username='slave'} at 127.20.107.136:33075
I20240503 08:13:06.205571 28525 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-320d1e880958427a88e93321171cb36e on tablet 320d1e880958427a88e93321171cb36e with peer b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:06.231640 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4ceeeb3495e4a8fac6e13e431c64af4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:06.239508 31771 ts_tablet_manager.cc:902] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:13:06.273563 31771 tablet_copy_client.cc:308] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:13:06.288367 31601 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:06.150967 31753 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:06.154417 31911 tablet_copy_client.cc:308] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:13:06.293762 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 4babf0ea9b954a23b1b1c4e403dce0bd from peer 21d21a5b596b4d48a17a1fb54a972844 ({username='slave'} at 127.20.107.134:58893)
I20240503 08:13:06.293947 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 4babf0ea9b954a23b1b1c4e403dce0bd from peer 21d21a5b596b4d48a17a1fb54a972844 at {username='slave'} at 127.20.107.134:58893: session id = 21d21a5b596b4d48a17a1fb54a972844-4babf0ea9b954a23b1b1c4e403dce0bd
I20240503 08:13:06.296587 28525 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet b0d07a6278d545e69bf7d2012109d1f4 from peer c8375219addc48fa99ac474ed1c9aef9 ({username='slave'} at 127.20.107.133:43991)
I20240503 08:13:06.296747 28525 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet b0d07a6278d545e69bf7d2012109d1f4 from peer c8375219addc48fa99ac474ed1c9aef9 at {username='slave'} at 127.20.107.133:43991: session id = c8375219addc48fa99ac474ed1c9aef9-b0d07a6278d545e69bf7d2012109d1f4
I20240503 08:13:06.297801 28785 tablet_copy_source_session.cc:205] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:06.299722 28525 tablet_copy_source_session.cc:205] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:06.299933 31911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4babf0ea9b954a23b1b1c4e403dce0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:06.301787 31771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d07a6278d545e69bf7d2012109d1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:06.302997 31464 raft_consensus.cc:1050] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER c8375219addc48fa99ac474ed1c9aef9 to VOTER
I20240503 08:13:06.303997 31464 consensus_queue.cc:237] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:06.306867 28373 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:06.307778 31753 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap complete.
I20240503 08:13:06.308097 31753 ts_tablet_manager.cc:1367] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844: Time spent bootstrapping tablet: real 1.155s user 0.052s sys 0.004s
I20240503 08:13:06.308490 31771 tablet_copy_client.cc:783] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:06.308701 31771 tablet_copy_client.cc:647] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:06.309631 31911 tablet_copy_client.cc:783] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:06.310032 31911 tablet_copy_client.cc:647] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:06.312075 31601 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93: Bootstrap complete.
I20240503 08:13:06.315601 30280 raft_consensus.cc:1261] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:06.315806 31771 tablet_copy_client.cc:523] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:06.316368 31504 consensus_queue.cc:1035] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:06.319554 28634 raft_consensus.cc:1261] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:06.319665 31771 tablet_bootstrap.cc:492] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap starting.
I20240503 08:13:06.320288 31504 consensus_queue.cc:1035] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:06.326185 31692 tablet_bootstrap.cc:492] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:06.326315 28501 raft_consensus.cc:1261] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:06.326912 31464 consensus_queue.cc:1035] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:06.329051 31650 ts_tablet_manager.cc:1398] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844: Time spent starting tablet: real 0.174s user 0.003s sys 0.001s
I20240503 08:13:06.330233 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session 21d21a5b596b4d48a17a1fb54a972844-b2ebb0b8475b4ed9a09711a87867de08 received from {username='slave'} at 127.20.107.134:32851
I20240503 08:13:06.330423 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session 21d21a5b596b4d48a17a1fb54a972844-b2ebb0b8475b4ed9a09711a87867de08 on tablet b2ebb0b8475b4ed9a09711a87867de08 with peer 21d21a5b596b4d48a17a1fb54a972844
I20240503 08:13:06.334617 31601 ts_tablet_manager.cc:1367] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93: Time spent bootstrapping tablet: real 1.318s user 0.060s sys 0.003s
I20240503 08:13:06.341500 31753 raft_consensus.cc:348] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:06.342132 31753 raft_consensus.cc:729] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Initialized, Role: LEARNER
I20240503 08:13:06.342398 31753 consensus_queue.cc:260] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:06.343176 31753 ts_tablet_manager.cc:1398] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844: Time spent starting tablet: real 0.035s user 0.003s sys 0.000s
I20240503 08:13:06.344082 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session 21d21a5b596b4d48a17a1fb54a972844-48e31346107a453a81921b23b244e7ff received from {username='slave'} at 127.20.107.134:32851
I20240503 08:13:06.344215 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session 21d21a5b596b4d48a17a1fb54a972844-48e31346107a453a81921b23b244e7ff on tablet 48e31346107a453a81921b23b244e7ff with peer 21d21a5b596b4d48a17a1fb54a972844
I20240503 08:13:06.346940 31828 tablet_bootstrap.cc:492] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:06.347373 31911 tablet_copy_client.cc:523] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:06.348009 31692 tablet_bootstrap.cc:492] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap complete.
I20240503 08:13:06.348735 31828 tablet_bootstrap.cc:492] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap complete.
I20240503 08:13:06.349051 31692 ts_tablet_manager.cc:1367] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent bootstrapping tablet: real 1.403s user 0.055s sys 0.008s
I20240503 08:13:06.352110 31692 raft_consensus.cc:348] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:06.353091 31692 raft_consensus.cc:729] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b29361eb4afa4ab3af7d8019d8a5042d, State: Initialized, Role: LEARNER
I20240503 08:13:06.353371 31692 consensus_queue.cc:260] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:06.356016 31692 ts_tablet_manager.cc:1398] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:13:06.356954 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-3bfa8537bc8a4950b53fbf3c36d5ecf3 received from {username='slave'} at 127.20.107.136:41343
I20240503 08:13:06.357115 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-3bfa8537bc8a4950b53fbf3c36d5ecf3 on tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3 with peer b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:06.360203 31828 ts_tablet_manager.cc:1367] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent bootstrapping tablet: real 0.856s user 0.049s sys 0.006s
I20240503 08:13:06.361274 31828 raft_consensus.cc:348] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:06.361562 31828 raft_consensus.cc:729] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b29361eb4afa4ab3af7d8019d8a5042d, State: Initialized, Role: LEARNER
I20240503 08:13:06.361778 31828 consensus_queue.cc:260] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:06.362638 31828 ts_tablet_manager.cc:1398] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:06.362699 31877 tablet_copy_client.cc:308] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:13:06.364946 28525 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-b0d07a6278d545e69bf7d2012109d1f4 received from {username='slave'} at 127.20.107.136:33075
I20240503 08:13:06.365119 28525 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-b0d07a6278d545e69bf7d2012109d1f4 on tablet b0d07a6278d545e69bf7d2012109d1f4 with peer b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:06.381196 31464 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:06.385462 28635 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:06.392601 31911 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap starting.
I20240503 08:13:06.393936 28228 catalog_manager.cc:5484] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:06.397069 28505 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:06.405977 28634 consensus_queue.cc:237] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:06.406529 31504 raft_consensus.cc:1050] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER c8375219addc48fa99ac474ed1c9aef9 to VOTER
I20240503 08:13:06.408023 31504 consensus_queue.cc:237] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:06.412678 28635 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:06.412681 30281 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:06.413396 31464 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:06.414443 30281 raft_consensus.cc:1203] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:06.415344 28505 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:06.415972 31464 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:06.418457 31504 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:06.421485 31597 ts_tablet_manager.cc:1398] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844: Time spent starting tablet: real 0.302s user 0.071s sys 0.046s
I20240503 08:13:06.422001 30281 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:06.422623 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session 21d21a5b596b4d48a17a1fb54a972844-0caf86850b3841ea811b0a6ab19ae330 received from {username='slave'} at 127.20.107.134:58893
I20240503 08:13:06.422791 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session 21d21a5b596b4d48a17a1fb54a972844-0caf86850b3841ea811b0a6ab19ae330 on tablet 0caf86850b3841ea811b0a6ab19ae330 with peer 21d21a5b596b4d48a17a1fb54a972844
I20240503 08:13:06.421343 31504 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:06.429145 28763 consensus_queue.cc:237] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:06.431600 30280 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:06.432765 28635 raft_consensus.cc:1261] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:06.434226 31954 consensus_queue.cc:1035] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:06.437206 28505 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:06.439383 28635 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:06.441908 30279 raft_consensus.cc:1261] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:06.442152 31464 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:06.451719 28225 catalog_manager.cc:5484] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:06.452885 28230 catalog_manager.cc:5484] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 reported cstate change: config changed from index 5 to 6, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:06.459370 28484 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e9537f30aaf45adbc2cc4a784009de7 with delete_type TABLET_DATA_TOMBSTONED (TS 716ffbb2a00b4f43b03e5cedcc9d302a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54162
I20240503 08:13:06.461611 28373 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:06.462494 31695 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:06.471917 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09156806b0744752969b6a574e24b50a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:06.476150 31954 consensus_queue.cc:1035] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20240503 08:13:06.485184 28700 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6e1b9f104ea471cab7527bd56433327. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:06.489552 28763 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:06.490978 31365 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:06.483456 31601 raft_consensus.cc:348] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:06.504097 31601 raft_consensus.cc:729] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Initialized, Role: LEARNER
I20240503 08:13:06.504845 31966 tablet_replica.cc:330] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a: stopping tablet replica
I20240503 08:13:06.504940 31601 consensus_queue.cc:260] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:06.505544 31966 raft_consensus.cc:2229] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:06.505913 31966 raft_consensus.cc:2258] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 08:13:06.506445 28698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.133:35121, user_credentials={real_user=slave}} blocked reactor thread for 62684.1us
I20240503 08:13:06.507647 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet c6e1b9f104ea471cab7527bd56433327 from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:43079)
I20240503 08:13:06.507900 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet c6e1b9f104ea471cab7527bd56433327 from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:43079: session id = bea2145b8e24436392b18ec3cfe90b93-c6e1b9f104ea471cab7527bd56433327
I20240503 08:13:06.509400 31365 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:06.510854 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e9537f30aaf45adbc2cc4a784009de7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:06.520627 31657 tablet_bootstrap.cc:492] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:06.521435 28228 catalog_manager.cc:5484] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:06.523303 31601 ts_tablet_manager.cc:1398] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93: Time spent starting tablet: real 0.188s user 0.012s sys 0.000s
I20240503 08:13:06.526392 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session bea2145b8e24436392b18ec3cfe90b93-48e31346107a453a81921b23b244e7ff received from {username='slave'} at 127.20.107.135:40941
I20240503 08:13:06.526556 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-48e31346107a453a81921b23b244e7ff on tablet 48e31346107a453a81921b23b244e7ff with peer bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:06.532502 31486 raft_consensus.cc:1050] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER c8375219addc48fa99ac474ed1c9aef9 to VOTER
I20240503 08:13:06.533957 31486 consensus_queue.cc:237] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:06.535478 31657 tablet_bootstrap.cc:492] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap complete.
I20240503 08:13:06.537377 30279 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:06.539831 31657 ts_tablet_manager.cc:1367] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93: Time spent bootstrapping tablet: real 1.589s user 0.064s sys 0.012s
I20240503 08:13:06.545333 28373 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:06.546531 31646 tablet_bootstrap.cc:492] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:06.547075 31646 tablet_bootstrap.cc:492] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap complete.
I20240503 08:13:06.547421 31646 ts_tablet_manager.cc:1367] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844: Time spent bootstrapping tablet: real 1.796s user 0.071s sys 0.007s
I20240503 08:13:06.542909 31657 raft_consensus.cc:348] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:06.548400 31657 raft_consensus.cc:729] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Initialized, Role: LEARNER
I20240503 08:13:06.548280 31646 raft_consensus.cc:348] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:06.548753 31646 raft_consensus.cc:729] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Initialized, Role: LEARNER
I20240503 08:13:06.548641 31657 consensus_queue.cc:260] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:06.553126 31601 ts_tablet_manager.cc:902] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:13:06.556970 31726 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:06.559084 31726 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93: Bootstrap complete.
I20240503 08:13:06.559384 31726 ts_tablet_manager.cc:1367] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93: Time spent bootstrapping tablet: real 1.541s user 0.060s sys 0.008s
I20240503 08:13:06.559736 30676 raft_consensus.cc:1203] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:13:06.560120 31726 raft_consensus.cc:348] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:06.560366 31726 raft_consensus.cc:729] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Initialized, Role: LEARNER
I20240503 08:13:06.560560 31726 consensus_queue.cc:260] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:06.564364 31966 ts_tablet_manager.cc:1875] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:06.553380 31657 ts_tablet_manager.cc:1398] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 08:13:06.567684 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session bea2145b8e24436392b18ec3cfe90b93-b2ebb0b8475b4ed9a09711a87867de08 received from {username='slave'} at 127.20.107.135:40941
I20240503 08:13:06.567821 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-b2ebb0b8475b4ed9a09711a87867de08 on tablet b2ebb0b8475b4ed9a09711a87867de08 with peer bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:06.576202 28634 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:06.582167 28763 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:06.549139 31646 consensus_queue.cc:260] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:06.589463 30414 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:06.590373 31488 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:06.590533 28785 tablet_copy_source_session.cc:205] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:06.593461 31646 ts_tablet_manager.cc:1398] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20240503 08:13:06.594285 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session 21d21a5b596b4d48a17a1fb54a972844-3bfa8537bc8a4950b53fbf3c36d5ecf3 received from {username='slave'} at 127.20.107.134:58893
I20240503 08:13:06.594419 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session 21d21a5b596b4d48a17a1fb54a972844-3bfa8537bc8a4950b53fbf3c36d5ecf3 on tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3 with peer 21d21a5b596b4d48a17a1fb54a972844
I20240503 08:13:06.594708 30279 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:06.596263 31488 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:06.596190 28762 consensus_queue.cc:237] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:06.601919 30280 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:06.602262 28634 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:06.601802 30676 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:06.603029 31504 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:06.603615 31959 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:06.605634 31726 ts_tablet_manager.cc:1398] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240503 08:13:06.608768 31959 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:06.611402 30280 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:06.612236 30543 raft_consensus.cc:1203] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:13:06.613539 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6e1b9f104ea471cab7527bd56433327 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:06.615748 30675 raft_consensus.cc:1203] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:06.616483 28229 catalog_manager.cc:5484] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 6 to 7, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:06.618386 28634 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:06.622862 28634 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:06.623332 31956 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:06.625555 31877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e1b9f104ea471cab7527bd56433327. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:13:06.626298 28700 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6e1b9f104ea471cab7527bd56433327. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:06.629210 30544 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:06.630436 31956 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:06.640132 31877 tablet_copy_client.cc:783] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:06.641492 31770 tablet_bootstrap.cc:492] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:06.642939 31770 tablet_bootstrap.cc:492] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap complete.
I20240503 08:13:06.643775 31886 tablet_bootstrap.cc:492] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:06.644217 31886 tablet_bootstrap.cc:492] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap complete.
I20240503 08:13:06.641731 31877 tablet_copy_client.cc:647] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:06.644526 31886 ts_tablet_manager.cc:1367] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d: Time spent bootstrapping tablet: real 0.695s user 0.049s sys 0.007s
I20240503 08:13:06.645381 31886 raft_consensus.cc:348] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:06.645646 31886 raft_consensus.cc:729] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b29361eb4afa4ab3af7d8019d8a5042d, State: Initialized, Role: LEARNER
I20240503 08:13:06.645974 31770 ts_tablet_manager.cc:1367] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93: Time spent bootstrapping tablet: real 0.951s user 0.055s sys 0.008s
I20240503 08:13:06.645869 31886 consensus_queue.cc:260] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:06.646625 31886 ts_tablet_manager.cc:1398] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 08:13:06.646322 31877 tablet_copy_client.cc:523] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:06.652165 31877 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
I20240503 08:13:06.653406 31770 raft_consensus.cc:348] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:06.653932 31770 raft_consensus.cc:729] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Initialized, Role: LEARNER
I20240503 08:13:06.654525 31486 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:06.656908 31770 consensus_queue.cc:260] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:06.657586 31770 ts_tablet_manager.cc:1398] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 08:13:06.656111 28635 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:06.658520 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session bea2145b8e24436392b18ec3cfe90b93-ede24a27608a4cf196dd46f85c4163e1 received from {username='slave'} at 127.20.107.135:40941
I20240503 08:13:06.658670 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-ede24a27608a4cf196dd46f85c4163e1 on tablet ede24a27608a4cf196dd46f85c4163e1 with peer bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:06.666677 31601 tablet_copy_client.cc:308] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:13:06.613621 28525 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session bea2145b8e24436392b18ec3cfe90b93-a20489bf65774902af38d228ee966ffe received from {username='slave'} at 127.20.107.135:59971
I20240503 08:13:06.629844 28484 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e1b9f104ea471cab7527bd56433327 with delete_type TABLET_DATA_TOMBSTONED (TS 716ffbb2a00b4f43b03e5cedcc9d302a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54162
I20240503 08:13:06.638348 28505 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:06.670653 28525 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-a20489bf65774902af38d228ee966ffe on tablet a20489bf65774902af38d228ee966ffe with peer bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:06.671152 28524 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-ac392ff7965a405f96efc777aca4f0bb received from {username='slave'} at 127.20.107.136:33075
I20240503 08:13:06.671303 28524 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-ac392ff7965a405f96efc777aca4f0bb on tablet ac392ff7965a405f96efc777aca4f0bb with peer b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:06.673286 30414 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:06.681130 31956 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:06.686450 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet b4ceeeb3495e4a8fac6e13e431c64af4 from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:40357)
I20240503 08:13:06.686656 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet b4ceeeb3495e4a8fac6e13e431c64af4 from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:40357: session id = bea2145b8e24436392b18ec3cfe90b93-b4ceeeb3495e4a8fac6e13e431c64af4
I20240503 08:13:06.695194 28655 tablet_copy_source_session.cc:205] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:06.698010 30413 raft_consensus.cc:1203] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:13:06.700943 28228 catalog_manager.cc:5484] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 6 to 7, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:06.704600 31601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ceeeb3495e4a8fac6e13e431c64af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:06.711493 28505 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:06.713450 30544 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:06.724109 30676 raft_consensus.cc:1203] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:06.735059 31966 ts_tablet_manager.cc:1888] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:06.737919 31601 tablet_copy_client.cc:783] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:06.738211 31601 tablet_copy_client.cc:647] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:06.738211 31966 log.cc:1196] T 6e9537f30aaf45adbc2cc4a784009de7 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/6e9537f30aaf45adbc2cc4a784009de7
I20240503 08:13:06.734745 31771 tablet_bootstrap.cc:492] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:06.742213 28218 catalog_manager.cc:4830] TS 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): tablet 6e9537f30aaf45adbc2cc4a784009de7 (table rebalance_test_table_9 [id=e4d3a510e86a4cbe8d3d611eac908187]) successfully deleted
I20240503 08:13:06.741948 30280 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:06.742853 31966 tablet_replica.cc:330] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a: stopping tablet replica
I20240503 08:13:06.743081 31966 raft_consensus.cc:2229] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:06.746058 31966 raft_consensus.cc:2258] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:06.747054 31771 tablet_bootstrap.cc:492] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap complete.
I20240503 08:13:06.747342 31771 ts_tablet_manager.cc:1367] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9: Time spent bootstrapping tablet: real 0.428s user 0.047s sys 0.007s
I20240503 08:13:06.749240 31966 ts_tablet_manager.cc:1875] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:06.749413 31771 raft_consensus.cc:348] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:06.749616 31863 raft_consensus.cc:1050] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 to VOTER
I20240503 08:13:06.750572 31863 consensus_queue.cc:237] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:06.749693 31771 raft_consensus.cc:729] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8375219addc48fa99ac474ed1c9aef9, State: Initialized, Role: LEARNER
I20240503 08:13:06.752183 31771 consensus_queue.cc:260] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:06.752746 31601 tablet_copy_client.cc:523] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:06.754685 31771 ts_tablet_manager.cc:1398] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:13:06.754987 30280 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:06.755436 28524 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session c8375219addc48fa99ac474ed1c9aef9-b0d07a6278d545e69bf7d2012109d1f4 received from {username='slave'} at 127.20.107.133:43991
I20240503 08:13:06.755563 28524 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session c8375219addc48fa99ac474ed1c9aef9-b0d07a6278d545e69bf7d2012109d1f4 on tablet b0d07a6278d545e69bf7d2012109d1f4 with peer c8375219addc48fa99ac474ed1c9aef9
I20240503 08:13:06.756480 30544 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:06.756431 31488 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:06.757165 31488 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:06.760954 28635 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:06.761478 31488 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:06.764602 28635 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.766185 30544 raft_consensus.cc:1203] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:06.775498 31488 raft_consensus.cc:1013] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: attempt to promote peer bea2145b8e24436392b18ec3cfe90b93: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:13:06.775761 31488 raft_consensus.cc:1013] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: attempt to promote peer bea2145b8e24436392b18ec3cfe90b93: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:13:06.776104 31488 raft_consensus.cc:1013] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: attempt to promote peer bea2145b8e24436392b18ec3cfe90b93: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:13:06.776257 31956 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.780493 28502 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:06.781881 31863 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:06.788273 28505 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.795512 30413 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:06.797331 31488 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240503 08:13:06.798646 30280 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.798669 30544 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.800385 31601 tablet_bootstrap.cc:492] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
I20240503 08:13:06.801610 30414 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.807139 31863 raft_consensus.cc:1050] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER bea2145b8e24436392b18ec3cfe90b93 to VOTER
I20240503 08:13:06.808208 31863 consensus_queue.cc:237] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:06.808503 28232 catalog_manager.cc:5484] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:06.811440 30544 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:06.811982 31488 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:06.812516 30280 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:06.813541 31956 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:06.822757 31966 ts_tablet_manager.cc:1888] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:13:06.825999 30412 raft_consensus.cc:1203] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:13:06.827128 31966 log.cc:1196] T c6e1b9f104ea471cab7527bd56433327 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/c6e1b9f104ea471cab7527bd56433327
I20240503 08:13:06.828989 28218 catalog_manager.cc:4830] TS 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): tablet c6e1b9f104ea471cab7527bd56433327 (table rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]) successfully deleted
I20240503 08:13:06.852007 30413 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:06.852634 28635 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:06.853284 31956 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:06.853740 31486 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:06.856132 31488 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.856664 28635 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.863521 30544 raft_consensus.cc:1203] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:06.865733 30413 raft_consensus.cc:1203] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:06.867539 30280 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.868098 30544 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.871212 28228 catalog_manager.cc:5484] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 8 to 9, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.863057 30414 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.879629 28505 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:06.881631 31863 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:06.883316 28502 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.888018 28373 consensus_queue.cc:237] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:06.893280 30280 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:06.894147 31486 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:06.894642 30544 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:06.901171 31996 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:06.907207 30542 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.913507 31486 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.918776 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b2ebb0b8475b4ed9a09711a87867de08 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:06.919409 30279 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.921185 30414 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:06.921772 31488 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:06.923557 30413 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.929790 28635 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:06.926885 28229 catalog_manager.cc:5484] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 reported cstate change: config changed from index 9 to 10, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.931314 31996 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:06.933214 28634 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.936175 28484 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ebb0b8475b4ed9a09711a87867de08 with delete_type TABLET_DATA_TOMBSTONED (TS 716ffbb2a00b4f43b03e5cedcc9d302a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:54162
I20240503 08:13:06.936558 31966 tablet_replica.cc:330] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a: stopping tablet replica
I20240503 08:13:06.936753 31966 raft_consensus.cc:2229] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:06.937278 31966 raft_consensus.cc:2258] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 08:13:06.940532 31486 consensus_queue.cc:1159] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4574396f35514c9caccb346f1a7af84d" responder_term: 1 status { last_received { term: 1 index: 10 } last_committed_idx: 10 last_received_current_leader { term: 1 index: 10 } }
I20240503 08:13:06.941715 28373 consensus_queue.cc:237] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:06.944629 30414 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:06.945418 31488 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:06.945866 31966 ts_tablet_manager.cc:1875] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:06.946942 30279 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:06.947702 31863 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:06.951861 31488 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.954641 31966 ts_tablet_manager.cc:1888] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:13:06.954842 31966 log.cc:1196] T b2ebb0b8475b4ed9a09711a87867de08 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/b2ebb0b8475b4ed9a09711a87867de08
I20240503 08:13:06.955721 28218 catalog_manager.cc:4830] TS 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): tablet b2ebb0b8475b4ed9a09711a87867de08 (table rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]) successfully deleted
I20240503 08:13:06.956350 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b2ebb0b8475b4ed9a09711a87867de08 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:06.957774 30414 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.960491 28232 catalog_manager.cc:5484] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 10 to 11, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:06.962529 30544 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:06.963269 31996 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:06.965142 28614 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ebb0b8475b4ed9a09711a87867de08 with delete_type TABLET_DATA_TOMBSTONED (TS 4574396f35514c9caccb346f1a7af84d not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:56340
I20240503 08:13:06.965324 30542 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.974018 31877 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:06.974644 31877 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap complete.
I20240503 08:13:06.974992 31877 ts_tablet_manager.cc:1367] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Time spent bootstrapping tablet: real 0.323s user 0.043s sys 0.024s
I20240503 08:13:06.974925 28373 consensus_queue.cc:237] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:06.978713 30414 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:13:06.978672 30279 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.979503 31996 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:13:06.982348 31488 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.983724 30280 raft_consensus.cc:1261] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:13:06.985597 30414 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.986737 31486 consensus_queue.cc:1035] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:13:06.988626 30280 raft_consensus.cc:2932] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.992153 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b2ebb0b8475b4ed9a09711a87867de08 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:06.995240 28232 catalog_manager.cc:5484] T b2ebb0b8475b4ed9a09711a87867de08 P c8375219addc48fa99ac474ed1c9aef9 reported cstate change: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:06.996604 31877 raft_consensus.cc:348] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:06.999907 31877 raft_consensus.cc:729] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Initialized, Role: LEARNER
I20240503 08:13:07.000725 31877 consensus_queue.cc:260] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:07.001694 31877 ts_tablet_manager.cc:1398] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Time spent starting tablet: real 0.027s user 0.003s sys 0.000s
I20240503 08:13:07.003597 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session bea2145b8e24436392b18ec3cfe90b93-c6e1b9f104ea471cab7527bd56433327 received from {username='slave'} at 127.20.107.135:43079
I20240503 08:13:07.003758 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-c6e1b9f104ea471cab7527bd56433327 on tablet c6e1b9f104ea471cab7527bd56433327 with peer bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:07.009243 28505 consensus_queue.cc:237] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:07.017939 31695 raft_consensus.cc:1050] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER c8375219addc48fa99ac474ed1c9aef9 to VOTER
I20240503 08:13:07.019315 31695 consensus_queue.cc:237] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
W20240503 08:13:07.035728 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 745d736d43244d4f9063f61de77cb473. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:07.035771 28635 raft_consensus.cc:1261] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:07.036857 28373 raft_consensus.cc:1261] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:07.037454 31886 ts_tablet_manager.cc:902] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:13:07.037674 32009 tablet_replica.cc:330] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d: stopping tablet replica
I20240503 08:13:07.038208 32009 raft_consensus.cc:2229] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:07.038230 31594 consensus_queue.cc:1035] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:07.039606 31594 consensus_queue.cc:1035] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:07.059770 31959 raft_consensus.cc:1050] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d to VOTER
I20240503 08:13:07.060716 31959 consensus_queue.cc:237] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:07.063690 32009 raft_consensus.cc:2258] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:07.065953 31833 raft_consensus.cc:1050] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d to VOTER
I20240503 08:13:07.067160 31833 consensus_queue.cc:237] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:07.076359 31539 raft_consensus.cc:1050] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d to VOTER
I20240503 08:13:07.076577 32009 ts_tablet_manager.cc:1875] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:07.077252 31539 consensus_queue.cc:237] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:07.082015 31877 tablet_replica.cc:330] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: stopping tablet replica
I20240503 08:13:07.082551 31877 raft_consensus.cc:2229] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 08:13:07.082895 31877 raft_consensus.cc:2258] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 08:13:07.084473 31877 ts_tablet_manager.cc:1875] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:07.095008 31911 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:07.097123 30523 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ebb0b8475b4ed9a09711a87867de08 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 11 (current committed config index is 12)) from {username='slave'} at 127.0.0.1:56760
I20240503 08:13:07.097522 31911 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap complete.
I20240503 08:13:07.097844 31911 ts_tablet_manager.cc:1367] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844: Time spent bootstrapping tablet: real 0.705s user 0.051s sys 0.007s
I20240503 08:13:07.101050 31911 raft_consensus.cc:348] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:07.103385 31911 raft_consensus.cc:729] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Initialized, Role: LEARNER
I20240503 08:13:07.103662 31911 consensus_queue.cc:260] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:07.104482 31911 ts_tablet_manager.cc:1398] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:13:07.105690 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session 21d21a5b596b4d48a17a1fb54a972844-4babf0ea9b954a23b1b1c4e403dce0bd received from {username='slave'} at 127.20.107.134:58893
I20240503 08:13:07.105854 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session 21d21a5b596b4d48a17a1fb54a972844-4babf0ea9b954a23b1b1c4e403dce0bd on tablet 4babf0ea9b954a23b1b1c4e403dce0bd with peer 21d21a5b596b4d48a17a1fb54a972844
I20240503 08:13:07.107170 30676 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.107291 30414 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.107808 28374 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.107977 31965 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.108515 31954 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.109134 31965 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.122864 31601 tablet_bootstrap.cc:492] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:07.125591 31601 tablet_bootstrap.cc:492] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap complete.
I20240503 08:13:07.126506 31601 ts_tablet_manager.cc:1367] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93: Time spent bootstrapping tablet: real 0.326s user 0.053s sys 0.003s
I20240503 08:13:07.126677 31504 raft_consensus.cc:1013] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: attempt to promote peer 21d21a5b596b4d48a17a1fb54a972844: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:13:07.134510 28633 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.135990 31504 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.141515 31877 ts_tablet_manager.cc:1888] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:13:07.141744 31877 log.cc:1196] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal/wals/c6e1b9f104ea471cab7527bd56433327
I20240503 08:13:07.142040 31488 raft_consensus.cc:1050] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER bea2145b8e24436392b18ec3cfe90b93 to VOTER
I20240503 08:13:07.142284 31877 ts_tablet_manager.cc:902] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:13:07.143144 31488 consensus_queue.cc:237] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:07.143415 32032 tablet_replica.cc:330] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93: stopping tablet replica
I20240503 08:13:07.143693 32032 raft_consensus.cc:2229] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:07.144345 32032 raft_consensus.cc:2258] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:07.146015 32032 ts_tablet_manager.cc:1875] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:07.129719 31601 raft_consensus.cc:348] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:07.148093 31601 raft_consensus.cc:729] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Initialized, Role: LEARNER
I20240503 08:13:07.148389 31601 consensus_queue.cc:260] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:07.153889 31601 ts_tablet_manager.cc:1398] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240503 08:13:07.173720 30542 raft_consensus.cc:1203] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:07.183780 31954 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:07.187274 30676 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:07.195451 30414 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:07.199079 28232 catalog_manager.cc:5484] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20240503 08:13:07.200465 31504 raft_consensus.cc:1050] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 to VOTER
I20240503 08:13:07.201339 31504 consensus_queue.cc:237] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:07.204602 30414 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:07.207301 31965 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:07.207115 32037 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:07.208679 28374 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:07.210137 31965 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.212566 28374 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:07.219760 30280 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.220551 31594 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.225303 28374 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.229048 31959 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
W20240503 08:13:07.233013 28439 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.129:38529, user_credentials={real_user=slave}} blocked reactor thread for 76843.2us
I20240503 08:13:07.234066 30674 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.234309 30673 raft_consensus.cc:1261] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.235656 28372 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.237059 28762 raft_consensus.cc:1261] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.237251 32047 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.237991 32047 consensus_queue.cc:1035] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.237954 32046 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.237645 28372 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:07.239131 32046 raft_consensus.cc:1050] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d to VOTER
I20240503 08:13:07.239748 32046 consensus_queue.cc:237] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:07.252187 32032 ts_tablet_manager.cc:1888] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240503 08:13:07.252352 32032 log.cc:1196] T b2ebb0b8475b4ed9a09711a87867de08 P bea2145b8e24436392b18ec3cfe90b93: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal/wals/b2ebb0b8475b4ed9a09711a87867de08
I20240503 08:13:07.254925 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session bea2145b8e24436392b18ec3cfe90b93-b4ceeeb3495e4a8fac6e13e431c64af4 received from {username='slave'} at 127.20.107.135:40357
I20240503 08:13:07.255097 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-b4ceeeb3495e4a8fac6e13e431c64af4 on tablet b4ceeeb3495e4a8fac6e13e431c64af4 with peer bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:07.257566 28763 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.258260 28762 raft_consensus.cc:1261] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.258759 30672 raft_consensus.cc:1261] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.260764 31539 consensus_queue.cc:1035] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 08:13:07.262065 31539 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.263841 31539 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 08:13:07.264798 31594 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.265177 28634 raft_consensus.cc:1261] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.266846 32046 consensus_queue.cc:1035] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.268664 28372 raft_consensus.cc:1261] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.265913 31539 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:07.270982 31965 raft_consensus.cc:1050] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 to VOTER
I20240503 08:13:07.272110 31965 consensus_queue.cc:237] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:07.275615 30279 raft_consensus.cc:1261] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.277326 30280 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:07.278450 31365 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.283699 28232 catalog_manager.cc:5484] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:07.286932 32009 ts_tablet_manager.cc:1888] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 08:13:07.287142 32009 log.cc:1196] T b2ebb0b8475b4ed9a09711a87867de08 P 4574396f35514c9caccb346f1a7af84d: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/b2ebb0b8475b4ed9a09711a87867de08
I20240503 08:13:07.290092 28763 raft_consensus.cc:1261] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.290804 31365 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.292129 31833 raft_consensus.cc:1050] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER c8375219addc48fa99ac474ed1c9aef9 to VOTER
I20240503 08:13:07.293099 31833 consensus_queue.cc:237] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:07.302489 28372 raft_consensus.cc:1261] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.303210 31365 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.303131 28217 catalog_manager.cc:4830] TS 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): tablet b2ebb0b8475b4ed9a09711a87867de08 (table rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]) successfully deleted
I20240503 08:13:07.304273 28219 catalog_manager.cc:4830] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): tablet b2ebb0b8475b4ed9a09711a87867de08 (table rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]) successfully deleted
I20240503 08:13:07.305142 30279 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:07.306079 31594 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:07.313037 30675 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:07.315049 32033 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:07.316917 28230 catalog_manager.cc:5484] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9 reported cstate change: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:07.317224 30675 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.317796 28635 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:07.325006 28232 catalog_manager.cc:5484] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 6 to 7, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.334160 28373 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:07.341723 30676 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:07.351969 31594 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.354458 31539 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:07.361578 30414 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.365852 30413 raft_consensus.cc:1203] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:07.367568 28634 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.367758 30413 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.369434 31863 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 08:13:07.370950 31863 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.376117 30542 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.376969 31863 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.378918 32045 raft_consensus.cc:1050] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER bea2145b8e24436392b18ec3cfe90b93 to VOTER
I20240503 08:13:07.379943 32045 consensus_queue.cc:237] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:13:07.392771 28228 catalog_manager.cc:5484] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:07.400461 31886 tablet_copy_client.cc:308] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:13:07.403014 28505 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.402525 31695 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.406330 28764 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.413667 31996 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.416546 31486 raft_consensus.cc:1050] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER bea2145b8e24436392b18ec3cfe90b93 to VOTER
I20240503 08:13:07.417594 31486 consensus_queue.cc:237] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:13:07.425791 28631 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:07.429347 28232 catalog_manager.cc:5484] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:07.430693 28371 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
W20240503 08:13:07.436563 28439 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.136:42389, user_credentials={real_user=slave}} blocked reactor thread for 82484.8us
I20240503 08:13:07.438899 28632 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:07.441079 31956 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:07.443786 32033 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.445086 28632 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:07.449774 30542 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:07.450803 28632 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.463404 28502 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:07.465967 30414 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.466136 28501 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.466691 31954 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.467320 31954 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.467983 28505 consensus_queue.cc:237] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:07.469412 28764 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:07.472290 28232 catalog_manager.cc:5484] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 5 to 6, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:07.477057 28635 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.479818 30281 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.482640 31504 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.485360 28762 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:07.493521 28761 raft_consensus.cc:1261] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.498481 30542 raft_consensus.cc:1261] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.499253 31863 consensus_queue.cc:1035] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.500299 31594 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:07.502035 30413 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:07.493378 28762 consensus_queue.cc:237] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:07.503352 31863 consensus_queue.cc:1035] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.506314 30673 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:07.484067 30675 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:07.507084 31959 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:07.507856 28232 catalog_manager.cc:5484] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:07.510315 32037 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.511013 28764 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:07.511229 30671 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:07.505553 30412 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:07.516207 28762 raft_consensus.cc:1261] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.516829 31504 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:07.519950 31504 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.521907 30412 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.522188 28762 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:07.522712 28632 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:07.523692 32043 consensus_queue.cc:1035] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.528003 28505 raft_consensus.cc:1261] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.531041 28501 consensus_queue.cc:237] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:07.531742 32043 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:07.529469 28230 catalog_manager.cc:5484] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: config changed from index 7 to 8, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.533777 32097 ts_tablet_manager.cc:902] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
W20240503 08:13:07.537427 32084 log.cc:924] Time spent T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20240503 08:13:07.538789 28761 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:07.541821 30671 raft_consensus.cc:1261] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.539295 28372 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:07.541095 28353 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3 with delete_type TABLET_DATA_TOMBSTONED (TS 4e4a1242f3714d03a6c9ee51dd63929c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:51022
I20240503 08:13:07.543609 32037 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.543943 31956 consensus_queue.cc:1035] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.544817 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 320d1e880958427a88e93321171cb36e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:07.547891 31996 raft_consensus.cc:1050] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 to VOTER
I20240503 08:13:07.549583 32037 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:07.549950 31996 consensus_queue.cc:237] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:07.550550 28230 catalog_manager.cc:5484] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:07.555249 30542 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:07.555573 28501 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:07.556172 31863 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:07.557274 31863 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:07.559417 31594 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
W20240503 08:13:07.560935 28218 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 48e31346107a453a81921b23b244e7ff with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:07.563483 28614 tablet_service.cc:1514] Processing DeleteTablet for tablet 320d1e880958427a88e93321171cb36e with delete_type TABLET_DATA_TOMBSTONED (TS 4574396f35514c9caccb346f1a7af84d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56340
I20240503 08:13:07.564572 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac392ff7965a405f96efc777aca4f0bb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:07.568600 30544 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.569254 28763 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.569922 32043 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.571148 31539 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.571688 31539 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.572203 32103 tablet_replica.cc:330] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c: stopping tablet replica
I20240503 08:13:07.572779 30671 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:07.536064 28370 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:07.577124 30674 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:07.577616 30281 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.579205 28227 catalog_manager.cc:5484] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d reported cstate change: config changed from index 5 to 6, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:07.581001 32043 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:07.582362 28630 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:07.583022 31488 raft_consensus.cc:2932] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.584652 30412 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:07.585521 31959 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:07.586122 31486 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:07.588146 32103 raft_consensus.cc:2229] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:07.588635 32103 raft_consensus.cc:2258] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:07.589592 28372 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.589697 28371 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.590396 30671 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.590873 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:07.592496 32087 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.594192 32043 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.595669 28759 consensus_queue.cc:237] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:07.598420 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet 09156806b0744752969b6a574e24b50a from peer b29361eb4afa4ab3af7d8019d8a5042d ({username='slave'} at 127.20.107.136:48299)
I20240503 08:13:07.598623 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet 09156806b0744752969b6a574e24b50a from peer b29361eb4afa4ab3af7d8019d8a5042d at {username='slave'} at 127.20.107.136:48299: session id = b29361eb4afa4ab3af7d8019d8a5042d-09156806b0744752969b6a574e24b50a
I20240503 08:13:07.597594 30412 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.600029 30672 raft_consensus.cc:1261] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.600764 32043 consensus_queue.cc:1035] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.602093 28655 tablet_copy_source_session.cc:205] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:07.602216 30675 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:07.603838 31886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09156806b0744752969b6a574e24b50a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.605320 30542 raft_consensus.cc:2932] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.608826 28353 tablet_service.cc:1514] Processing DeleteTablet for tablet ac392ff7965a405f96efc777aca4f0bb with delete_type TABLET_DATA_TOMBSTONED (TS 4e4a1242f3714d03a6c9ee51dd63929c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51022
I20240503 08:13:07.611709 28635 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:07.612509 32086 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:07.613934 30672 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.614356 28635 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.620082 32009 tablet_replica.cc:330] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d: stopping tablet replica
I20240503 08:13:07.620047 28630 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.620626 32009 raft_consensus.cc:2229] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:07.620857 32009 raft_consensus.cc:2258] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:07.622112 32009 ts_tablet_manager.cc:1875] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:07.629912 30543 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.631778 32009 ts_tablet_manager.cc:1888] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:07.631934 32009 log.cc:1196] T 320d1e880958427a88e93321171cb36e P 4574396f35514c9caccb346f1a7af84d: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/320d1e880958427a88e93321171cb36e
I20240503 08:13:07.635243 28631 consensus_queue.cc:237] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:07.637020 28501 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.637670 30278 raft_consensus.cc:1261] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.637614 28765 raft_consensus.cc:2932] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.638309 31976 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.640928 31976 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.641698 28505 raft_consensus.cc:2932] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.645984 30278 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
W20240503 08:13:07.648507 28218 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 48e31346107a453a81921b23b244e7ff with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 6 but the committed config has opid_index of 7
I20240503 08:13:07.648914 28217 catalog_manager.cc:4830] TS 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): tablet 320d1e880958427a88e93321171cb36e (table rebalance_test_table_5 [id=311fe8a0590b44a8a07f0203d8f65e32]) successfully deleted
I20240503 08:13:07.650694 28230 catalog_manager.cc:5484] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 6 to 7, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:07.654142 32103 ts_tablet_manager.cc:1875] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:07.661078 28227 catalog_manager.cc:5484] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 reported cstate change: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.661406 28229 catalog_manager.cc:5484] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93 reported cstate change: config changed from index 6 to 7, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.662918 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 585ba864cc384e7f9db90e9d4c47e82c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:07.664697 28760 raft_consensus.cc:1261] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.669728 31954 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.673089 31886 tablet_copy_client.cc:783] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:07.674922 31886 tablet_copy_client.cc:647] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:07.679024 28632 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.684764 28744 tablet_service.cc:1514] Processing DeleteTablet for tablet 585ba864cc384e7f9db90e9d4c47e82c with delete_type TABLET_DATA_TOMBSTONED (TS 93f7e733cec9415eb422b0a7ba8621cb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39918
I20240503 08:13:07.685027 31886 tablet_copy_client.cc:523] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:07.686208 28229 catalog_manager.cc:5484] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93 reported cstate change: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.692799 31886 tablet_bootstrap.cc:492] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap starting.
I20240503 08:13:07.695026 30413 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:07.696378 30671 raft_consensus.cc:1261] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:07.699798 31954 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:07.699851 30413 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.705196 28501 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.706961 28228 catalog_manager.cc:5484] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.718839 28372 consensus_queue.cc:237] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:07.717582 28761 consensus_queue.cc:237] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:07.720001 31877 tablet_copy_client.cc:308] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:13:07.720750 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet c6e1b9f104ea471cab7527bd56433327 from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:43079)
I20240503 08:13:07.720953 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet c6e1b9f104ea471cab7527bd56433327 from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:43079: session id = bea2145b8e24436392b18ec3cfe90b93-c6e1b9f104ea471cab7527bd56433327
I20240503 08:13:07.721382 31504 consensus_queue.cc:1035] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:07.722075 30413 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:07.723395 30542 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:07.723893 28785 tablet_copy_source_session.cc:205] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:07.724130 31996 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:07.724730 31996 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:07.730522 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:07.730952 31996 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.733453 31954 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.738441 30542 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.741459 31877 ts_tablet_manager.cc:1875] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:13:07.741693 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:07.743455 28229 catalog_manager.cc:5484] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 8 to 9, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:07.747318 28228 catalog_manager.cc:5484] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93 reported cstate change: config changed from index 7 to 8, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.747256 30413 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.748924 28501 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:07.749213 30412 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.749622 32087 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:07.750375 31965 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.751200 28502 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.752835 30412 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.757733 28372 raft_consensus.cc:1261] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.758064 28614 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e31346107a453a81921b23b244e7ff with delete_type TABLET_DATA_TOMBSTONED (TS 4574396f35514c9caccb346f1a7af84d not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:56340
I20240503 08:13:07.758589 32009 tablet_replica.cc:330] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: stopping tablet replica
I20240503 08:13:07.758886 32009 raft_consensus.cc:2229] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:07.758857 31365 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.759255 32009 raft_consensus.cc:2258] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:07.760603 28372 consensus_queue.cc:237] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:13:07.761243 31539 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.761256 28371 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.762679 32037 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.763600 30542 raft_consensus.cc:1261] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.764230 31996 consensus_queue.cc:1035] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.766410 28762 raft_consensus.cc:1261] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.767022 31996 consensus_queue.cc:1035] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.767845 28501 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.769308 31863 raft_consensus.cc:2932] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.771282 32009 ts_tablet_manager.cc:1875] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:07.776551 30542 raft_consensus.cc:2932] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.781354 31965 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.781874 28762 raft_consensus.cc:2932] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.782526 32033 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.786311 30543 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.786851 28763 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:07.786142 30413 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.789019 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0caf86850b3841ea811b0a6ab19ae330 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:07.789322 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48e31346107a453a81921b23b244e7ff with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:07.789566 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ede24a27608a4cf196dd46f85c4163e1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:07.792426 28225 catalog_manager.cc:5484] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:07.792641 28371 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.795979 30671 raft_consensus.cc:2932] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.799302 28502 consensus_queue.cc:237] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:13:07.800484 31365 raft_consensus.cc:1050] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER bea2145b8e24436392b18ec3cfe90b93 to VOTER
I20240503 08:13:07.801432 31365 consensus_queue.cc:237] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:13:07.802165 28228 catalog_manager.cc:5484] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:07.804522 30543 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.805222 32045 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.806460 28761 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.808882 28502 consensus_queue.cc:237] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:07.813179 30281 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:07.813903 31833 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:07.814563 28372 consensus_queue.cc:237] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:07.815100 28373 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:07.816391 30675 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:07.816190 31833 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.820959 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a20489bf65774902af38d228ee966ffe with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:07.821291 32123 tablet_replica.cc:330] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb: stopping tablet replica
I20240503 08:13:07.821677 32045 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:07.822528 32045 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:07.823778 28232 catalog_manager.cc:5484] T ede24a27608a4cf196dd46f85c4163e1 P bea2145b8e24436392b18ec3cfe90b93 reported cstate change: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.823959 30543 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.828312 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:07.828501 28614 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3 with delete_type TABLET_DATA_TOMBSTONED (TS 4574396f35514c9caccb346f1a7af84d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56340
I20240503 08:13:07.837180 28502 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.837540 32037 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.843616 32123 raft_consensus.cc:2229] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:07.847257 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b0d07a6278d545e69bf7d2012109d1f4 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:07.849218 28371 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:07.847368 32033 raft_consensus.cc:1050] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 to VOTER
I20240503 08:13:07.851281 32033 consensus_queue.cc:237] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:07.852174 32045 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:07.859261 28371 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.862614 28614 tablet_service.cc:1514] Processing DeleteTablet for tablet 0caf86850b3841ea811b0a6ab19ae330 with delete_type TABLET_DATA_TOMBSTONED (TS 4574396f35514c9caccb346f1a7af84d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56340
I20240503 08:13:07.863900 28373 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.826340 28762 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.870775 28484 tablet_service.cc:1514] Processing DeleteTablet for tablet ede24a27608a4cf196dd46f85c4163e1 with delete_type TABLET_DATA_TOMBSTONED (TS 716ffbb2a00b4f43b03e5cedcc9d302a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54162
I20240503 08:13:07.874485 30281 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.877014 28230 catalog_manager.cc:5484] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.879861 28372 consensus_queue.cc:237] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:07.883044 28228 catalog_manager.cc:5484] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9 reported cstate change: config changed from index 7 to 8, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.885747 28744 tablet_service.cc:1514] Processing DeleteTablet for tablet a20489bf65774902af38d228ee966ffe with delete_type TABLET_DATA_TOMBSTONED (TS 93f7e733cec9415eb422b0a7ba8621cb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39918
I20240503 08:13:07.888546 30543 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.889344 31976 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.893278 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:07.894567 32123 raft_consensus.cc:2258] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:07.900162 28373 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.901207 28502 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.901934 31365 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.902547 31976 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.907004 32097 tablet_copy_client.cc:308] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:13:07.908221 30413 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.908315 28505 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.908733 28524 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet 745d736d43244d4f9063f61de77cb473 from peer c8375219addc48fa99ac474ed1c9aef9 ({username='slave'} at 127.20.107.133:43991)
I20240503 08:13:07.908958 28524 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet 745d736d43244d4f9063f61de77cb473 from peer c8375219addc48fa99ac474ed1c9aef9 at {username='slave'} at 127.20.107.133:43991: session id = c8375219addc48fa99ac474ed1c9aef9-745d736d43244d4f9063f61de77cb473
I20240503 08:13:07.909116 30542 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:07.910354 30671 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.911954 28763 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:07.912448 28524 tablet_copy_source_session.cc:205] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:07.913311 30413 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:07.913323 31488 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:07.915496 32097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745d736d43244d4f9063f61de77cb473. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.916505 28505 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:07.917570 31954 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.919322 32009 ts_tablet_manager.cc:1888] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:13:07.919502 32009 log.cc:1196] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/48e31346107a453a81921b23b244e7ff
I20240503 08:13:07.917858 32087 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.921990 32086 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:07.923153 32086 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:07.923671 31996 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:07.925452 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48e31346107a453a81921b23b244e7ff with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:07.926110 30413 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.927845 31954 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:07.929142 28761 consensus_queue.cc:237] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:07.929536 28217 catalog_manager.cc:4830] TS 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): tablet 48e31346107a453a81921b23b244e7ff (table rebalance_test_table_9 [id=e4d3a510e86a4cbe8d3d611eac908187]) successfully deleted
I20240503 08:13:07.930500 31695 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.935200 30544 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:07.935654 28372 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.941362 32142 tablet_replica.cc:330] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: stopping tablet replica
I20240503 08:13:07.941599 32142 raft_consensus.cc:2229] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:07.941859 32142 raft_consensus.cc:2258] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:07.942101 32097 tablet_copy_client.cc:783] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:07.942080 30542 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.942389 32097 tablet_copy_client.cc:647] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:07.943259 32142 ts_tablet_manager.cc:1875] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:07.944401 28635 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:07.945226 32009 tablet_replica.cc:330] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d: stopping tablet replica
I20240503 08:13:07.945438 32009 raft_consensus.cc:2229] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:07.945751 32009 raft_consensus.cc:2258] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:07.947170 32097 tablet_copy_client.cc:523] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:07.948761 31954 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 08:13:07.950970 28231 catalog_manager.cc:5484] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 8 to 9, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:07.955708 31996 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:07.963842 28225 catalog_manager.cc:5484] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:07.964040 28762 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:07.972199 28225 catalog_manager.cc:5484] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:07.974581 28744 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d07a6278d545e69bf7d2012109d1f4 with delete_type TABLET_DATA_TOMBSTONED (TS 93f7e733cec9415eb422b0a7ba8621cb not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:39918
I20240503 08:13:07.975353 32097 tablet_bootstrap.cc:492] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap starting.
I20240503 08:13:07.965174 28484 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e31346107a453a81921b23b244e7ff with delete_type TABLET_DATA_TOMBSTONED (TS 716ffbb2a00b4f43b03e5cedcc9d302a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:54162
I20240503 08:13:07.976930 28504 consensus_queue.cc:237] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:07.979218 30671 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:07.980026 30281 raft_consensus.cc:1261] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:07.980705 32037 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:07.981877 28504 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:07.985302 30278 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.985600 31539 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:07.990568 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b0d07a6278d545e69bf7d2012109d1f4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:07.992147 32045 consensus_queue.cc:1035] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:07.996773 31877 ts_tablet_manager.cc:1888] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:13:07.997210 31877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e1b9f104ea471cab7527bd56433327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.998260 30671 raft_consensus.cc:2932] T b0d07a6278d545e69bf7d2012109d1f4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:08.002947 28502 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
W20240503 08:13:08.005066 28700 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:08.011298 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6e1b9f104ea471cab7527bd56433327. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.013276 30281 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:08.013507 32123 ts_tablet_manager.cc:1875] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:08.017179 32009 ts_tablet_manager.cc:1875] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:08.024056 32103 ts_tablet_manager.cc:1888] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:13:08.024355 32103 log.cc:1196] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/3bfa8537bc8a4950b53fbf3c36d5ecf3
I20240503 08:13:08.025460 32103 tablet_replica.cc:330] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c: stopping tablet replica
I20240503 08:13:08.028106 28631 consensus_queue.cc:237] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:13:08.030689 30542 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:08.031268 31695 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:08.031991 28218 catalog_manager.cc:4830] TS 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3 (table rebalance_test_table_0 [id=c2cf824bda7e41fb9ad499d46ed7c4d1]) successfully deleted
I20240503 08:13:08.032004 31954 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20240503 08:13:08.033545 28630 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:08.034112 31695 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:08.037604 28505 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:08.038969 31959 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:08.041169 30543 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:08.045970 30413 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:08.051051 28631 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:08.054674 32009 ts_tablet_manager.cc:1888] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
W20240503 08:13:08.028950 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9997e8da99c4f45a63bc3db521b33ba. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.056463 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:08.046347 28371 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:08.059255 32103 raft_consensus.cc:2229] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:08.059618 32103 raft_consensus.cc:2258] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:08.054356 28224 catalog_manager.cc:5484] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:08.062155 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b4ceeeb3495e4a8fac6e13e431c64af4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:08.074829 31959 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:08.076136 28484 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ceeeb3495e4a8fac6e13e431c64af4 with delete_type TABLET_DATA_TOMBSTONED (TS 716ffbb2a00b4f43b03e5cedcc9d302a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54162
I20240503 08:13:08.077153 32123 ts_tablet_manager.cc:1888] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:08.077347 32123 log.cc:1196] T 585ba864cc384e7f9db90e9d4c47e82c P 93f7e733cec9415eb422b0a7ba8621cb: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/wals/585ba864cc384e7f9db90e9d4c47e82c
I20240503 08:13:08.078357 32123 tablet_replica.cc:330] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb: stopping tablet replica
I20240503 08:13:08.078563 32123 raft_consensus.cc:2229] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:08.057821 31365 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:08.082124 28371 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:08.089704 32009 log.cc:1196] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 4574396f35514c9caccb346f1a7af84d: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/3bfa8537bc8a4950b53fbf3c36d5ecf3
I20240503 08:13:08.090888 32009 tablet_replica.cc:330] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: stopping tablet replica
I20240503 08:13:08.092654 32103 ts_tablet_manager.cc:1875] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:08.095266 32009 raft_consensus.cc:2229] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:08.095642 32009 raft_consensus.cc:2258] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:08.095939 32123 raft_consensus.cc:2258] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:08.098981 31877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.20.107.132:36209 (request call id 4) took 1020 ms. Trace:
I20240503 08:13:08.099568 28218 catalog_manager.cc:4830] TS 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095): tablet 585ba864cc384e7f9db90e9d4c47e82c (table rebalance_test_table_6 [id=87e8ca2152ac403cb71e5eb67c0b2e0e]) successfully deleted
I20240503 08:13:08.100555 28217 catalog_manager.cc:4830] TS 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3 (table rebalance_test_table_0 [id=c2cf824bda7e41fb9ad499d46ed7c4d1]) successfully deleted
I20240503 08:13:08.101013 28228 catalog_manager.cc:5484] T b0d07a6278d545e69bf7d2012109d1f4 P c8375219addc48fa99ac474ed1c9aef9 reported cstate change: config changed from index 8 to 9, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:08.101887 32123 ts_tablet_manager.cc:1875] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:08.102097 32009 ts_tablet_manager.cc:1875] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:08.109411 28225 catalog_manager.cc:5484] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:08.110963 28353 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d07a6278d545e69bf7d2012109d1f4 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 9)) from {username='slave'} at 127.0.0.1:51022
I20240503 08:13:08.099153 31877 rpcz_store.cc:276] 0503 08:13:07.078086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 08:13:07.081678 (+ 3592us) service_pool.cc:225] Handling call
0503 08:13:07.087879 (+ 6201us) tablet_metadata.cc:641] Metadata flushed
0503 08:13:07.141327 (+ 53448us) tablet_metadata.cc:641] Metadata flushed
0503 08:13:07.142226 (+ 899us) ts_tablet_manager.cc:888] Shutdown and deleted data from running replica
0503 08:13:07.142327 (+ 101us) ts_tablet_manager.cc:903] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
0503 08:13:07.741128 (+598801us) tablet_copy_client.cc:353] Tablet copy session begun
0503 08:13:07.745235 (+ 4107us) tablet_metadata.cc:641] Metadata flushed
0503 08:13:07.996577 (+251342us) tablet_metadata.cc:641] Metadata flushed
0503 08:13:07.997064 (+ 487us) tablet_copy_client.cc:448] Replaced tombstoned tablet metadata.
0503 08:13:08.098570 (+101506us) tablet_copy_client.cc:403] Wrote new consensus metadata
0503 08:13:08.098740 (+ 170us) tablet_replica.cc:205] Creating consensus instance
0503 08:13:08.098882 (+ 142us) ts_tablet_manager.cc:958] Replica registered.
0503 08:13:08.098912 (+ 30us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":85,"tablet-copy.queue_time_us":58,"thread_start_us":577237,"threads_started":4}
I20240503 08:13:08.111691 31877 tablet_copy_client.cc:783] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:08.111971 31877 tablet_copy_client.cc:647] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:08.113176 30281 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:08.121497 28227 catalog_manager.cc:5484] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 reported cstate change: config changed from index 7 to 8, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:08.124045 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:08.124701 32142 ts_tablet_manager.cc:1888] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:08.124908 32142 log.cc:1196] T ede24a27608a4cf196dd46f85c4163e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/ede24a27608a4cf196dd46f85c4163e1
I20240503 08:13:08.125778 32142 tablet_replica.cc:330] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: stopping tablet replica
I20240503 08:13:08.126042 32142 raft_consensus.cc:2229] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:08.126565 28218 catalog_manager.cc:4830] TS 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): tablet ede24a27608a4cf196dd46f85c4163e1 (table rebalance_test_table_0 [id=c2cf824bda7e41fb9ad499d46ed7c4d1]) successfully deleted
I20240503 08:13:08.130761 31877 tablet_copy_client.cc:523] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:08.126453 32142 raft_consensus.cc:2258] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:08.144675 32142 ts_tablet_manager.cc:1875] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:08.150008 32103 ts_tablet_manager.cc:1888] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:08.107409 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:08.110785 32033 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:08.152994 32103 log.cc:1196] T ac392ff7965a405f96efc777aca4f0bb P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/ac392ff7965a405f96efc777aca4f0bb
I20240503 08:13:08.153862 28218 catalog_manager.cc:4830] TS 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): tablet ac392ff7965a405f96efc777aca4f0bb (table rebalance_test_table_2 [id=2ff6aeab7e1847f6884750db73f8c428]) successfully deleted
I20240503 08:13:08.154450 32103 tablet_replica.cc:330] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c: stopping tablet replica
I20240503 08:13:08.154731 32103 raft_consensus.cc:2229] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:08.155263 32103 raft_consensus.cc:2258] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:08.133247 28761 consensus_queue.cc:237] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:08.158676 32103 ts_tablet_manager.cc:1875] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:08.163672 31877 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
I20240503 08:13:08.165015 30413 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:08.166250 28505 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:08.167181 31954 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:08.170905 31965 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:08.174808 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4babf0ea9b954a23b1b1c4e403dce0bd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:08.176499 30413 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:08.179761 32097 tablet_bootstrap.cc:492] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:08.180284 32097 tablet_bootstrap.cc:492] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap complete.
I20240503 08:13:08.180603 32097 ts_tablet_manager.cc:1367] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9: Time spent bootstrapping tablet: real 0.205s user 0.053s sys 0.008s
I20240503 08:13:08.182260 31965 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:08.184190 28505 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:08.186703 28227 catalog_manager.cc:5484] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:08.194751 30523 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:08.181506 32097 raft_consensus.cc:348] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:08.205526 32097 raft_consensus.cc:729] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8375219addc48fa99ac474ed1c9aef9, State: Initialized, Role: LEARNER
I20240503 08:13:08.205811 32097 consensus_queue.cc:260] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:08.206470 32097 ts_tablet_manager.cc:1398] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240503 08:13:08.207659 28614 tablet_service.cc:1514] Processing DeleteTablet for tablet 4babf0ea9b954a23b1b1c4e403dce0bd with delete_type TABLET_DATA_TOMBSTONED (TS 4574396f35514c9caccb346f1a7af84d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56340
I20240503 08:13:08.208253 32103 ts_tablet_manager.cc:1888] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:08.208401 32103 log.cc:1196] T b0d07a6278d545e69bf7d2012109d1f4 P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/b0d07a6278d545e69bf7d2012109d1f4
I20240503 08:13:08.210444 32009 ts_tablet_manager.cc:1888] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:08.210587 32009 log.cc:1196] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/0caf86850b3841ea811b0a6ab19ae330
I20240503 08:13:08.211131 28218 catalog_manager.cc:4830] TS 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): tablet b0d07a6278d545e69bf7d2012109d1f4 (table rebalance_test_table_4 [id=3a90dbe17357489db92bd286b71aa565]) successfully deleted
I20240503 08:13:08.211303 28217 catalog_manager.cc:4830] TS 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): tablet 0caf86850b3841ea811b0a6ab19ae330 (table rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]) successfully deleted
I20240503 08:13:08.217420 28524 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session c8375219addc48fa99ac474ed1c9aef9-745d736d43244d4f9063f61de77cb473 received from {username='slave'} at 127.20.107.133:43991
I20240503 08:13:08.217589 28524 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session c8375219addc48fa99ac474ed1c9aef9-745d736d43244d4f9063f61de77cb473 on tablet 745d736d43244d4f9063f61de77cb473 with peer c8375219addc48fa99ac474ed1c9aef9
I20240503 08:13:08.219141 28631 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:08.224298 32142 ts_tablet_manager.cc:1888] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:08.244223 32123 ts_tablet_manager.cc:1888] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:08.258392 32142 log.cc:1196] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/48e31346107a453a81921b23b244e7ff
I20240503 08:13:08.259596 28218 catalog_manager.cc:4830] TS 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): tablet 48e31346107a453a81921b23b244e7ff (table rebalance_test_table_9 [id=e4d3a510e86a4cbe8d3d611eac908187]) successfully deleted
I20240503 08:13:08.260263 32142 tablet_replica.cc:330] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: stopping tablet replica
I20240503 08:13:08.260625 32142 raft_consensus.cc:2229] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:08.261215 32009 tablet_replica.cc:330] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: stopping tablet replica
I20240503 08:13:08.261515 32009 raft_consensus.cc:2229] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:08.261878 32009 raft_consensus.cc:2258] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:08.271036 32009 ts_tablet_manager.cc:1875] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:08.276721 32142 raft_consensus.cc:2258] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:08.278204 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:08.278553 32142 ts_tablet_manager.cc:1875] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:08.293402 28505 consensus_queue.cc:237] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:08.297039 32123 log.cc:1196] T a20489bf65774902af38d228ee966ffe P 93f7e733cec9415eb422b0a7ba8621cb: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/wals/a20489bf65774902af38d228ee966ffe
I20240503 08:13:08.298056 32123 tablet_replica.cc:330] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb: stopping tablet replica
I20240503 08:13:08.298306 32123 raft_consensus.cc:2229] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:08.298761 32123 raft_consensus.cc:2258] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:08.299890 28218 catalog_manager.cc:4830] TS 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095): tablet a20489bf65774902af38d228ee966ffe (table rebalance_test_table_6 [id=87e8ca2152ac403cb71e5eb67c0b2e0e]) successfully deleted
I20240503 08:13:08.303745 28631 consensus_queue.cc:237] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:08.313570 32123 ts_tablet_manager.cc:1875] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 08:13:08.315750 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5736ee8065344ea8124fdf759035216. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.320649 32180 ts_tablet_manager.cc:902] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:08.321754 28761 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:08.322896 31539 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:08.325795 28630 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:08.326591 31539 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:08.357189 32123 ts_tablet_manager.cc:1888] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:13:08.358808 31539 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:08.367792 28232 catalog_manager.cc:5484] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:13:08.380168 28570 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d113e8de02f447b9da0a1cb03686cd2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.382053 32009 ts_tablet_manager.cc:1888] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:08.382360 28371 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:08.361871 28763 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:08.397058 32123 log.cc:1196] T b0d07a6278d545e69bf7d2012109d1f4 P 93f7e733cec9415eb422b0a7ba8621cb: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/wals/b0d07a6278d545e69bf7d2012109d1f4
I20240503 08:13:08.383671 28761 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:08.402694 32142 ts_tablet_manager.cc:1888] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:08.405699 28218 catalog_manager.cc:4830] TS 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095): tablet b0d07a6278d545e69bf7d2012109d1f4 (table rebalance_test_table_4 [id=3a90dbe17357489db92bd286b71aa565]) successfully deleted
I20240503 08:13:08.383056 31365 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:08.413758 32164 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:08.398902 28630 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:08.440735 32009 log.cc:1196] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/4babf0ea9b954a23b1b1c4e403dce0bd
I20240503 08:13:08.442075 28217 catalog_manager.cc:4830] TS 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): tablet 4babf0ea9b954a23b1b1c4e403dce0bd (table rebalance_test_table_5 [id=311fe8a0590b44a8a07f0203d8f65e32]) successfully deleted
I20240503 08:13:08.450851 32142 log.cc:1196] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/b4ceeeb3495e4a8fac6e13e431c64af4
I20240503 08:13:08.452230 28218 catalog_manager.cc:4830] TS 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): tablet b4ceeeb3495e4a8fac6e13e431c64af4 (table rebalance_test_table_7 [id=0225804a5c4a42cbb32c4b0029752bd1]) successfully deleted
I20240503 08:13:08.468945 32164 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:08.478178 28372 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:08.512245 28631 consensus_queue.cc:237] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:08.501961 28761 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:08.491205 28228 catalog_manager.cc:5484] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
W20240503 08:13:08.521400 28570 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d113e8de02f447b9da0a1cb03686cd2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.537266 31886 tablet_bootstrap.cc:492] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:08.542498 31886 tablet_bootstrap.cc:492] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap complete.
I20240503 08:13:08.544987 31886 ts_tablet_manager.cc:1367] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d: Time spent bootstrapping tablet: real 0.852s user 0.053s sys 0.008s
I20240503 08:13:08.548180 31886 raft_consensus.cc:348] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:08.549400 31886 raft_consensus.cc:729] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b29361eb4afa4ab3af7d8019d8a5042d, State: Initialized, Role: LEARNER
I20240503 08:13:08.549707 31886 consensus_queue.cc:260] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:08.556245 31886 ts_tablet_manager.cc:1398] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:13:08.558163 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-09156806b0744752969b6a574e24b50a received from {username='slave'} at 127.20.107.136:48299
I20240503 08:13:08.558343 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-09156806b0744752969b6a574e24b50a on tablet 09156806b0744752969b6a574e24b50a with peer b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:08.572590 28371 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:08.574196 28762 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:08.577131 32164 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:08.580906 32164 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:08.595669 32180 tablet_copy_client.cc:308] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:08.597970 31976 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:08.610589 28228 catalog_manager.cc:5484] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:08.612072 30281 raft_consensus.cc:1203] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:08.613893 31886 ts_tablet_manager.cc:902] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:13:08.647274 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet e9997e8da99c4f45a63bc3db521b33ba from peer b29361eb4afa4ab3af7d8019d8a5042d ({username='slave'} at 127.20.107.136:56655)
I20240503 08:13:08.647472 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet e9997e8da99c4f45a63bc3db521b33ba from peer b29361eb4afa4ab3af7d8019d8a5042d at {username='slave'} at 127.20.107.136:56655: session id = b29361eb4afa4ab3af7d8019d8a5042d-e9997e8da99c4f45a63bc3db521b33ba
W20240503 08:13:08.648600 28569 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.132:34095, user_credentials={real_user=slave}} blocked reactor thread for 50948.9us
I20240503 08:13:08.650187 30671 raft_consensus.cc:1203] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:08.650354 28394 tablet_copy_source_session.cc:205] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:08.650940 28762 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:08.652446 32180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9997e8da99c4f45a63bc3db521b33ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.661536 32180 tablet_copy_client.cc:783] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:08.665452 32180 tablet_copy_client.cc:647] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:08.671553 32180 tablet_copy_client.cc:523] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:08.691812 32180 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap starting.
I20240503 08:13:08.705054 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:08.730974 28371 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
W20240503 08:13:08.746881 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d113e8de02f447b9da0a1cb03686cd2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.759755 32247 ts_tablet_manager.cc:902] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:13:08.771216 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:08.783699 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:08.792909 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:08.807775 31886 tablet_copy_client.cc:308] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:13:08.808900 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet c6e1b9f104ea471cab7527bd56433327 from peer b29361eb4afa4ab3af7d8019d8a5042d ({username='slave'} at 127.20.107.136:41343)
I20240503 08:13:08.809111 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet c6e1b9f104ea471cab7527bd56433327 from peer b29361eb4afa4ab3af7d8019d8a5042d at {username='slave'} at 127.20.107.136:41343: session id = b29361eb4afa4ab3af7d8019d8a5042d-c6e1b9f104ea471cab7527bd56433327
I20240503 08:13:08.812000 28785 tablet_copy_source_session.cc:205] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:08.814298 31886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e1b9f104ea471cab7527bd56433327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.819579 28371 consensus_queue.cc:237] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:08.821475 32247 tablet_copy_client.cc:308] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:13:08.822312 28524 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet b5736ee8065344ea8124fdf759035216 from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:59971)
I20240503 08:13:08.822484 28524 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet b5736ee8065344ea8124fdf759035216 from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:59971: session id = bea2145b8e24436392b18ec3cfe90b93-b5736ee8065344ea8124fdf759035216
I20240503 08:13:08.824714 28505 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:08.825332 32086 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:08.825585 28524 tablet_copy_source_session.cc:205] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 08:13:08.826138 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:08.826431 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9997e8da99c4f45a63bc3db521b33ba. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.829020 28762 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:08.829141 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:08.829811 31863 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:08.830786 30523 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:08.837396 28224 catalog_manager.cc:5484] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:08.837831 28505 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:08.855201 32247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5736ee8065344ea8124fdf759035216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.857844 31863 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:08.860006 28762 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:08.863870 31886 tablet_copy_client.cc:783] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:08.864169 31886 tablet_copy_client.cc:647] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:08.885361 32247 tablet_copy_client.cc:783] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:08.886574 32247 tablet_copy_client.cc:647] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:08.889029 28371 consensus_queue.cc:237] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:08.890805 32247 tablet_copy_client.cc:523] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:08.894732 32247 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
I20240503 08:13:08.896942 28631 consensus_queue.cc:237] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:08.909162 28372 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:08.909940 32230 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:08.914748 28630 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:08.915462 32087 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:08.918176 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d13c33b8712e4c488abb99b73ff42b86. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.929374 31886 tablet_copy_client.cc:523] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 08:13:08.930330 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d60a162155504975a39c20eff88341e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.930919 28505 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:08.931723 32230 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:08.943074 28761 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:08.948568 31863 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:08.951196 31886 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap starting.
I20240503 08:13:08.954313 31976 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:08.957937 28502 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:08.966037 28228 catalog_manager.cc:5484] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:08.972939 28631 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:08.989812 32087 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:09.025947 32180 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:09.033526 28371 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:09.049201 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:09.026486 32180 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap complete.
I20240503 08:13:09.055651 32180 ts_tablet_manager.cc:1367] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d: Time spent bootstrapping tablet: real 0.364s user 0.049s sys 0.011s
I20240503 08:13:09.055651 31539 raft_consensus.cc:1050] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER c8375219addc48fa99ac474ed1c9aef9 to VOTER
I20240503 08:13:09.056650 31539 consensus_queue.cc:237] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:09.065008 32180 raft_consensus.cc:348] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:09.065342 32180 raft_consensus.cc:729] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b29361eb4afa4ab3af7d8019d8a5042d, State: Initialized, Role: LEARNER
I20240503 08:13:09.065569 32180 consensus_queue.cc:260] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:09.069519 32180 ts_tablet_manager.cc:1398] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 08:13:09.070343 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-e9997e8da99c4f45a63bc3db521b33ba received from {username='slave'} at 127.20.107.136:56655
I20240503 08:13:09.070518 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-e9997e8da99c4f45a63bc3db521b33ba on tablet e9997e8da99c4f45a63bc3db521b33ba with peer b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:09.081110 30281 raft_consensus.cc:1261] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:09.081890 32037 consensus_queue.cc:1035] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:09.085243 28228 catalog_manager.cc:5484] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:09.089164 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:09.089892 28630 raft_consensus.cc:1261] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:09.090238 28762 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:09.096536 30523 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:09.097035 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:09.119376 31365 raft_consensus.cc:1050] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d to VOTER
I20240503 08:13:09.119938 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:09.120935 31365 consensus_queue.cc:237] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:09.133649 32304 ts_tablet_manager.cc:902] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:13:09.135206 30671 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:09.136173 31976 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:09.140852 28761 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:09.141610 31976 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:09.142547 28372 consensus_queue.cc:237] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:09.149428 28373 raft_consensus.cc:1261] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:09.152577 28373 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:09.153349 32230 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:09.163722 31539 consensus_queue.cc:1035] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20240503 08:13:09.164741 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:09.164992 32045 consensus_queue.cc:1035] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 08:13:09.167945 28761 raft_consensus.cc:1261] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:09.168362 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8a9adc3712d436c95505a3216e7607c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:09.168978 32275 consensus_queue.cc:1035] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.173987 28635 consensus_queue.cc:237] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:09.181210 28502 raft_consensus.cc:1261] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.182528 32275 consensus_queue.cc:1035] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:09.190089 28568 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d44335d8be7245b1bb52b2eab9e42c18. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:09.192299 28505 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.193508 31365 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:09.195163 31976 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.199136 30675 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:09.209391 32086 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:09.215101 28761 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:09.219588 28372 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.221534 31976 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.222162 31976 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
W20240503 08:13:09.225965 32303 log.cc:924] Time spent T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20240503 08:13:09.232954 32305 ts_tablet_manager.cc:902] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:13:09.242743 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:09.248018 28504 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:09.257720 28504 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:09.258093 28763 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:09.260402 28373 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:09.261821 32045 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:09.270203 30281 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:09.273375 32304 tablet_copy_client.cc:308] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:13:09.273687 28371 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:09.285706 28224 catalog_manager.cc:5484] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d reported cstate change: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:09.274191 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet 2d113e8de02f447b9da0a1cb03686cd2 from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:40357)
I20240503 08:13:09.289814 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet 2d113e8de02f447b9da0a1cb03686cd2 from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:40357: session id = bea2145b8e24436392b18ec3cfe90b93-2d113e8de02f447b9da0a1cb03686cd2
I20240503 08:13:09.266742 28635 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:09.294557 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:09.296131 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:09.309597 30671 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:09.311110 32337 ts_tablet_manager.cc:902] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:09.313026 28655 tablet_copy_source_session.cc:205] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:09.318746 31886 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:09.320142 31886 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap complete.
I20240503 08:13:09.320597 31886 ts_tablet_manager.cc:1367] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent bootstrapping tablet: real 0.370s user 0.064s sys 0.000s
I20240503 08:13:09.322356 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:09.324163 32304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d113e8de02f447b9da0a1cb03686cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.331348 28371 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:09.332376 31886 raft_consensus.cc:348] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:09.333333 31886 raft_consensus.cc:729] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b29361eb4afa4ab3af7d8019d8a5042d, State: Initialized, Role: LEARNER
I20240503 08:13:09.336803 32337 tablet_copy_client.cc:308] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:09.343382 32305 tablet_copy_client.cc:308] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:13:09.358851 28504 consensus_queue.cc:237] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:09.364835 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet d13c33b8712e4c488abb99b73ff42b86 from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:40941)
I20240503 08:13:09.365782 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet d13c33b8712e4c488abb99b73ff42b86 from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:40941: session id = bea2145b8e24436392b18ec3cfe90b93-d13c33b8712e4c488abb99b73ff42b86
I20240503 08:13:09.362386 28232 catalog_manager.cc:5484] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9 reported cstate change: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:09.375542 28394 tablet_copy_source_session.cc:205] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:09.376379 28231 catalog_manager.cc:5484] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:09.377261 32337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d13c33b8712e4c488abb99b73ff42b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.385586 32337 tablet_copy_client.cc:783] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:09.385881 32337 tablet_copy_client.cc:647] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:09.407141 28231 catalog_manager.cc:5484] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:09.414561 28631 consensus_queue.cc:237] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:09.418427 28505 consensus_queue.cc:237] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:09.422416 30281 raft_consensus.cc:1261] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:09.423063 32316 consensus_queue.cc:1035] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:09.425648 32316 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:09.425348 31886 consensus_queue.cc:260] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:09.427496 32337 tablet_copy_client.cc:523] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:09.428807 32340 ts_tablet_manager.cc:902] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:09.429970 30278 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:09.436419 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 745d736d43244d4f9063f61de77cb473 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:09.440650 28762 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:09.440490 28231 catalog_manager.cc:5484] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 5 to 6, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:09.441524 32328 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:09.445350 28353 tablet_service.cc:1514] Processing DeleteTablet for tablet 745d736d43244d4f9063f61de77cb473 with delete_type TABLET_DATA_TOMBSTONED (TS 4e4a1242f3714d03a6c9ee51dd63929c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51022
I20240503 08:13:09.448472 28630 raft_consensus.cc:1261] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:09.448616 31976 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:09.449903 32316 consensus_queue.cc:1035] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:09.451586 28630 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:09.453744 28762 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:09.454883 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 09156806b0744752969b6a574e24b50a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:09.458844 28227 catalog_manager.cc:5484] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 5 to 6, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:09.464290 28354 tablet_service.cc:1514] Processing DeleteTablet for tablet 09156806b0744752969b6a574e24b50a with delete_type TABLET_DATA_TOMBSTONED (TS 4e4a1242f3714d03a6c9ee51dd63929c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51022
I20240503 08:13:09.473220 31877 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:09.474045 31877 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap complete.
I20240503 08:13:09.474378 31877 ts_tablet_manager.cc:1367] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Time spent bootstrapping tablet: real 1.311s user 0.063s sys 0.008s
I20240503 08:13:09.487488 32304 tablet_copy_client.cc:783] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:09.487954 32304 tablet_copy_client.cc:647] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:09.505808 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet 2d113e8de02f447b9da0a1cb03686cd2 from peer 21d21a5b596b4d48a17a1fb54a972844 ({username='slave'} at 127.20.107.134:52703)
I20240503 08:13:09.506007 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet 2d113e8de02f447b9da0a1cb03686cd2 from peer 21d21a5b596b4d48a17a1fb54a972844 at {username='slave'} at 127.20.107.134:52703: session id = 21d21a5b596b4d48a17a1fb54a972844-2d113e8de02f447b9da0a1cb03686cd2
I20240503 08:13:09.507663 32337 tablet_bootstrap.cc:492] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
I20240503 08:13:09.510098 28655 tablet_copy_source_session.cc:205] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:09.511747 32305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d113e8de02f447b9da0a1cb03686cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.513005 28635 raft_consensus.cc:1261] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.513025 30671 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:09.513598 31886 ts_tablet_manager.cc:1398] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent starting tablet: real 0.106s user 0.001s sys 0.001s
I20240503 08:13:09.513593 32315 consensus_queue.cc:1035] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.513757 31976 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:09.514901 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-c6e1b9f104ea471cab7527bd56433327 received from {username='slave'} at 127.20.107.136:41343
I20240503 08:13:09.515062 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-c6e1b9f104ea471cab7527bd56433327 on tablet c6e1b9f104ea471cab7527bd56433327 with peer b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:09.515620 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8b4e398d34443b4a440f9ba48555d84. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:09.516638 30671 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:09.517796 28763 raft_consensus.cc:1261] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.518596 32315 consensus_queue.cc:1035] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.518743 32305 tablet_copy_client.cc:783] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:09.518994 32305 tablet_copy_client.cc:647] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:09.521780 32305 tablet_copy_client.cc:523] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:09.526443 32305 tablet_bootstrap.cc:492] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap starting.
I20240503 08:13:09.532068 31539 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:09.532706 28762 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:09.538550 28227 catalog_manager.cc:5484] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:09.541980 31886 ts_tablet_manager.cc:902] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:13:09.469151 32349 tablet_replica.cc:330] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: stopping tablet replica
I20240503 08:13:09.543560 32349 raft_consensus.cc:2229] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:09.551870 32349 raft_consensus.cc:2258] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:09.553562 32349 ts_tablet_manager.cc:1875] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:09.557468 28635 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:09.579505 32304 tablet_copy_client.cc:523] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:09.583577 28504 consensus_queue.cc:237] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
W20240503 08:13:09.591647 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5736ee8065344ea8124fdf759035216. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:09.592217 31886 tablet_copy_client.cc:308] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:13:09.592957 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet d60a162155504975a39c20eff88341e1 from peer b29361eb4afa4ab3af7d8019d8a5042d ({username='slave'} at 127.20.107.136:48299)
I20240503 08:13:09.593149 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet d60a162155504975a39c20eff88341e1 from peer b29361eb4afa4ab3af7d8019d8a5042d at {username='slave'} at 127.20.107.136:48299: session id = b29361eb4afa4ab3af7d8019d8a5042d-d60a162155504975a39c20eff88341e1
I20240503 08:13:09.594203 28763 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:13:09.595695 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:09.596088 31539 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:09.597218 28655 tablet_copy_source_session.cc:205] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:09.598623 28630 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:09.598829 31886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60a162155504975a39c20eff88341e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.599336 31539 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:09.601233 32045 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:09.601238 28763 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:09.609311 32304 tablet_bootstrap.cc:492] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
I20240503 08:13:09.610245 28631 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:09.617563 28227 catalog_manager.cc:5484] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 4 to 5, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:09.617969 31877 raft_consensus.cc:348] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:09.618237 31877 raft_consensus.cc:729] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Initialized, Role: LEARNER
I20240503 08:13:09.618500 31877 consensus_queue.cc:260] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:09.658537 28371 consensus_queue.cc:237] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:09.682971 32086 raft_consensus.cc:1050] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d to VOTER
I20240503 08:13:09.684134 32086 consensus_queue.cc:237] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:09.691324 32374 ts_tablet_manager.cc:902] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:13:09.692021 28763 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 08:13:09.693060 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9997e8da99c4f45a63bc3db521b33ba. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:09.694252 30671 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:09.694679 31863 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:09.695888 32275 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:09.698014 28504 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:09.698522 32086 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:09.699102 30671 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:09.709470 28226 catalog_manager.cc:5484] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d reported cstate change: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:09.723275 32087 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:09.723462 28635 consensus_queue.cc:237] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:09.726583 28763 raft_consensus.cc:1261] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.727201 28374 raft_consensus.cc:1261] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.727901 32328 consensus_queue.cc:1035] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:09.728261 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28e42c3244f949c7b7343bf587894ac5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:09.727957 28763 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:09.728644 32328 consensus_queue.cc:1035] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.742645 32374 tablet_copy_client.cc:308] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:13:09.743647 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet d44335d8be7245b1bb52b2eab9e42c18 from peer c8375219addc48fa99ac474ed1c9aef9 ({username='slave'} at 127.20.107.133:44309)
I20240503 08:13:09.743803 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet d44335d8be7245b1bb52b2eab9e42c18 from peer c8375219addc48fa99ac474ed1c9aef9 at {username='slave'} at 127.20.107.133:44309: session id = c8375219addc48fa99ac474ed1c9aef9-d44335d8be7245b1bb52b2eab9e42c18
I20240503 08:13:09.744206 28504 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:09.745477 31976 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:09.747205 28655 tablet_copy_source_session.cc:205] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:09.749274 32374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44335d8be7245b1bb52b2eab9e42c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.749682 28373 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:09.762441 28226 catalog_manager.cc:5484] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:09.780913 32377 ts_tablet_manager.cc:902] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:09.782729 28761 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.787636 31886 tablet_copy_client.cc:783] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:09.788017 31886 tablet_copy_client.cc:647] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:09.789711 31886 tablet_copy_client.cc:523] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:09.794848 31886 tablet_bootstrap.cc:492] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap starting.
I20240503 08:13:09.800778 32374 tablet_copy_client.cc:783] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:09.801575 32374 tablet_copy_client.cc:647] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:09.804319 32374 tablet_copy_client.cc:523] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:09.811691 30671 raft_consensus.cc:1203] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:13:09.783444 32086 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20240503 08:13:09.827148 31877 ts_tablet_manager.cc:1398] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Time spent starting tablet: real 0.353s user 0.003s sys 0.002s
I20240503 08:13:09.827831 28763 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:09.831144 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session bea2145b8e24436392b18ec3cfe90b93-c6e1b9f104ea471cab7527bd56433327 received from {username='slave'} at 127.20.107.135:43079
I20240503 08:13:09.831305 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-c6e1b9f104ea471cab7527bd56433327 on tablet c6e1b9f104ea471cab7527bd56433327 with peer bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:09.833949 32374 tablet_bootstrap.cc:492] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap starting.
I20240503 08:13:09.778815 32349 ts_tablet_manager.cc:1888] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:09.835143 32349 log.cc:1196] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/745d736d43244d4f9063f61de77cb473
I20240503 08:13:09.836457 32349 tablet_replica.cc:330] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: stopping tablet replica
I20240503 08:13:09.836512 28218 catalog_manager.cc:4830] TS 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): tablet 745d736d43244d4f9063f61de77cb473 (table rebalance_test_table_0 [id=c2cf824bda7e41fb9ad499d46ed7c4d1]) successfully deleted
W20240503 08:13:09.844586 28308 consensus_peers.cc:485] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b3b8b80ebd44f27899f97211266c7b2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:09.848034 28504 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.860972 32275 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.863736 32340 tablet_copy_client.cc:308] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:09.864635 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet e9997e8da99c4f45a63bc3db521b33ba from peer 21d21a5b596b4d48a17a1fb54a972844 ({username='slave'} at 127.20.107.134:32851)
I20240503 08:13:09.864821 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet e9997e8da99c4f45a63bc3db521b33ba from peer 21d21a5b596b4d48a17a1fb54a972844 at {username='slave'} at 127.20.107.134:32851: session id = 21d21a5b596b4d48a17a1fb54a972844-e9997e8da99c4f45a63bc3db521b33ba
I20240503 08:13:09.866370 32349 raft_consensus.cc:2229] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:09.866639 32349 raft_consensus.cc:2258] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:09.867869 32349 ts_tablet_manager.cc:1875] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:09.868335 28394 tablet_copy_source_session.cc:205] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:09.870054 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9997e8da99c4f45a63bc3db521b33ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.872120 32086 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:09.877100 28505 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:09.880363 28226 catalog_manager.cc:5484] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:09.881677 28762 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:09.906505 32305 tablet_bootstrap.cc:492] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:09.907035 32305 tablet_bootstrap.cc:492] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap complete.
I20240503 08:13:09.907331 32305 ts_tablet_manager.cc:1367] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844: Time spent bootstrapping tablet: real 0.381s user 0.056s sys 0.009s
I20240503 08:13:09.909322 32305 raft_consensus.cc:348] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:09.910002 32305 raft_consensus.cc:729] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Initialized, Role: LEARNER
I20240503 08:13:09.910238 32305 consensus_queue.cc:260] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:09.919426 32340 tablet_copy_client.cc:783] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:09.965626 32377 tablet_copy_client.cc:308] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:09.973156 32427 ts_tablet_manager.cc:902] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:13:09.973759 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet a8a9adc3712d436c95505a3216e7607c from peer c8375219addc48fa99ac474ed1c9aef9 ({username='slave'} at 127.20.107.133:49403)
I20240503 08:13:09.967857 32305 ts_tablet_manager.cc:1398] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844: Time spent starting tablet: real 0.060s user 0.012s sys 0.005s
I20240503 08:13:09.981669 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session 21d21a5b596b4d48a17a1fb54a972844-2d113e8de02f447b9da0a1cb03686cd2 received from {username='slave'} at 127.20.107.134:52703
I20240503 08:13:09.981818 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session 21d21a5b596b4d48a17a1fb54a972844-2d113e8de02f447b9da0a1cb03686cd2 on tablet 2d113e8de02f447b9da0a1cb03686cd2 with peer 21d21a5b596b4d48a17a1fb54a972844
I20240503 08:13:09.982007 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet a8a9adc3712d436c95505a3216e7607c from peer c8375219addc48fa99ac474ed1c9aef9 at {username='slave'} at 127.20.107.133:49403: session id = c8375219addc48fa99ac474ed1c9aef9-a8a9adc3712d436c95505a3216e7607c
I20240503 08:13:09.985157 32340 tablet_copy_client.cc:647] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:09.985428 28394 tablet_copy_source_session.cc:205] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:09.987105 32337 tablet_bootstrap.cc:492] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:09.987847 32337 tablet_bootstrap.cc:492] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap complete.
I20240503 08:13:09.988394 32337 ts_tablet_manager.cc:1367] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93: Time spent bootstrapping tablet: real 0.481s user 0.058s sys 0.003s
I20240503 08:13:09.989231 32337 raft_consensus.cc:348] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:09.989559 32337 raft_consensus.cc:729] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Initialized, Role: LEARNER
I20240503 08:13:09.989796 32337 consensus_queue.cc:260] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:09.987483 32377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a9adc3712d436c95505a3216e7607c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.990644 32340 tablet_copy_client.cc:523] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:09.991472 32337 ts_tablet_manager.cc:1398] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:13:09.992280 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session bea2145b8e24436392b18ec3cfe90b93-d13c33b8712e4c488abb99b73ff42b86 received from {username='slave'} at 127.20.107.135:40941
I20240503 08:13:09.995579 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-d13c33b8712e4c488abb99b73ff42b86 on tablet d13c33b8712e4c488abb99b73ff42b86 with peer bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:10.002815 32349 ts_tablet_manager.cc:1888] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:10.008932 32349 log.cc:1196] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/09156806b0744752969b6a574e24b50a
I20240503 08:13:10.014326 32340 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap starting.
I20240503 08:13:10.017966 28218 catalog_manager.cc:4830] TS 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): tablet 09156806b0744752969b6a574e24b50a (table rebalance_test_table_4 [id=3a90dbe17357489db92bd286b71aa565]) successfully deleted
I20240503 08:13:10.026515 32377 tablet_copy_client.cc:783] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:10.027850 32377 tablet_copy_client.cc:647] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:10.032730 32377 tablet_copy_client.cc:523] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:10.050259 32377 tablet_bootstrap.cc:492] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9: Bootstrap starting.
I20240503 08:13:10.051805 31886 tablet_bootstrap.cc:492] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:10.052277 31886 tablet_bootstrap.cc:492] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap complete.
I20240503 08:13:10.052561 31886 ts_tablet_manager.cc:1367] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent bootstrapping tablet: real 0.258s user 0.051s sys 0.010s
I20240503 08:13:10.056885 31886 raft_consensus.cc:348] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:10.057181 31886 raft_consensus.cc:729] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b29361eb4afa4ab3af7d8019d8a5042d, State: Initialized, Role: LEARNER
I20240503 08:13:10.057413 31886 consensus_queue.cc:260] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:10.070392 30413 raft_consensus.cc:1203] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:13:10.077939 32442 ts_tablet_manager.cc:902] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:13:10.079630 31886 ts_tablet_manager.cc:1398] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent starting tablet: real 0.027s user 0.002s sys 0.003s
I20240503 08:13:10.080446 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-d60a162155504975a39c20eff88341e1 received from {username='slave'} at 127.20.107.136:48299
I20240503 08:13:10.080650 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-d60a162155504975a39c20eff88341e1 on tablet d60a162155504975a39c20eff88341e1 with peer b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:10.091109 32442 tablet_copy_client.cc:308] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:13:10.092252 28524 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet b5736ee8065344ea8124fdf759035216 from peer b29361eb4afa4ab3af7d8019d8a5042d ({username='slave'} at 127.20.107.136:33075)
I20240503 08:13:10.092449 28524 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet b5736ee8065344ea8124fdf759035216 from peer b29361eb4afa4ab3af7d8019d8a5042d at {username='slave'} at 127.20.107.136:33075: session id = b29361eb4afa4ab3af7d8019d8a5042d-b5736ee8065344ea8124fdf759035216
I20240503 08:13:10.095902 28524 tablet_copy_source_session.cc:205] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:10.097476 32442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5736ee8065344ea8124fdf759035216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:10.099457 30671 raft_consensus.cc:1203] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:10.105573 32442 tablet_copy_client.cc:783] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:10.106047 32442 tablet_copy_client.cc:647] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:10.108265 32442 tablet_copy_client.cc:523] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:10.112944 32442 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap starting.
I20240503 08:13:10.114928 32247 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:10.115406 32247 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap complete.
I20240503 08:13:10.115702 32247 ts_tablet_manager.cc:1367] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93: Time spent bootstrapping tablet: real 1.221s user 0.059s sys 0.004s
I20240503 08:13:10.116514 32247 raft_consensus.cc:348] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:10.150701 32247 raft_consensus.cc:729] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Initialized, Role: LEARNER
I20240503 08:13:10.150298 30542 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:10.151005 32247 consensus_queue.cc:260] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:10.151932 32247 ts_tablet_manager.cc:1398] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93: Time spent starting tablet: real 0.036s user 0.003s sys 0.000s
I20240503 08:13:10.157095 32427 tablet_copy_client.cc:308] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:13:10.158023 28524 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet a8b4e398d34443b4a440f9ba48555d84 from peer c8375219addc48fa99ac474ed1c9aef9 ({username='slave'} at 127.20.107.133:43991)
I20240503 08:13:10.158193 28524 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet a8b4e398d34443b4a440f9ba48555d84 from peer c8375219addc48fa99ac474ed1c9aef9 at {username='slave'} at 127.20.107.133:43991: session id = c8375219addc48fa99ac474ed1c9aef9-a8b4e398d34443b4a440f9ba48555d84
I20240503 08:13:10.170176 28371 consensus_queue.cc:237] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:10.176651 28762 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:10.176946 30671 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:10.177937 32086 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:10.179262 28505 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:10.183791 28525 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session bea2145b8e24436392b18ec3cfe90b93-b5736ee8065344ea8124fdf759035216 received from {username='slave'} at 127.20.107.135:59971
I20240503 08:13:10.183966 28525 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-b5736ee8065344ea8124fdf759035216 on tablet b5736ee8065344ea8124fdf759035216 with peer bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:10.184480 32087 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:10.185154 32087 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:10.193717 32395 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.193958 28505 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.198279 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e9997e8da99c4f45a63bc3db521b33ba with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:10.198593 28762 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.199389 30671 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.204052 28524 tablet_copy_source_session.cc:205] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:10.205703 32427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b4e398d34443b4a440f9ba48555d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:10.207360 32442 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:10.207834 32442 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap complete.
I20240503 08:13:10.207826 28226 catalog_manager.cc:5484] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d reported cstate change: config changed from index 6 to 7, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.208766 32442 ts_tablet_manager.cc:1367] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent bootstrapping tablet: real 0.096s user 0.055s sys 0.019s
I20240503 08:13:10.210932 32442 raft_consensus.cc:348] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:10.211264 32442 raft_consensus.cc:729] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b29361eb4afa4ab3af7d8019d8a5042d, State: Initialized, Role: LEARNER
I20240503 08:13:10.211489 32442 consensus_queue.cc:260] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:10.212170 32442 ts_tablet_manager.cc:1398] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:13:10.213558 28524 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-b5736ee8065344ea8124fdf759035216 received from {username='slave'} at 127.20.107.136:33075
I20240503 08:13:10.213747 28524 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-b5736ee8065344ea8124fdf759035216 on tablet b5736ee8065344ea8124fdf759035216 with peer b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:10.213872 28371 consensus_queue.cc:237] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:10.216281 30671 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:10.216926 32275 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:10.220296 32275 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.225317 30671 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.235679 32442 ts_tablet_manager.cc:902] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:10.246412 28226 catalog_manager.cc:5484] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 7 to 8, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:10.253270 28504 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:10.254276 32275 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:10.256170 28504 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.277434 32305 ts_tablet_manager.cc:902] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:13:10.280263 32305 tablet_copy_client.cc:308] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:13:10.281227 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet 28e42c3244f949c7b7343bf587894ac5 from peer 21d21a5b596b4d48a17a1fb54a972844 ({username='slave'} at 127.20.107.134:52703)
I20240503 08:13:10.281472 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet 28e42c3244f949c7b7343bf587894ac5 from peer 21d21a5b596b4d48a17a1fb54a972844 at {username='slave'} at 127.20.107.134:52703: session id = 21d21a5b596b4d48a17a1fb54a972844-28e42c3244f949c7b7343bf587894ac5
I20240503 08:13:10.285218 28655 tablet_copy_source_session.cc:205] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:10.287209 32305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e42c3244f949c7b7343bf587894ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:10.298274 30543 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:10.300048 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e9997e8da99c4f45a63bc3db521b33ba with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:10.300333 28744 tablet_service.cc:1514] Processing DeleteTablet for tablet e9997e8da99c4f45a63bc3db521b33ba with delete_type TABLET_DATA_TOMBSTONED (TS 93f7e733cec9415eb422b0a7ba8621cb not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:39918
I20240503 08:13:10.308385 32305 tablet_copy_client.cc:783] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:10.310031 32305 tablet_copy_client.cc:647] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:10.311254 32471 tablet_replica.cc:330] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb: stopping tablet replica
I20240503 08:13:10.311507 32471 raft_consensus.cc:2229] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:10.311961 32471 raft_consensus.cc:2258] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:10.313304 32427 tablet_copy_client.cc:783] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:10.313446 32305 tablet_copy_client.cc:523] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:10.313634 32427 tablet_copy_client.cc:647] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:10.317277 32427 tablet_copy_client.cc:523] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:10.322549 32382 raft_consensus.cc:1050] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d to VOTER
I20240503 08:13:10.323499 32382 consensus_queue.cc:237] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:10.333639 32305 tablet_bootstrap.cc:492] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap starting.
I20240503 08:13:10.333627 32471 ts_tablet_manager.cc:1875] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:10.349150 30671 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:10.349990 31965 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:10.351018 30278 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:10.357333 32481 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:10.359056 28635 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:10.360139 32382 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:10.361179 30543 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:10.365547 32481 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:10.371277 30542 raft_consensus.cc:1203] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:10.393323 31965 raft_consensus.cc:1013] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: attempt to promote peer bea2145b8e24436392b18ec3cfe90b93: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:13:10.403651 30394 tablet_service.cc:1514] Processing DeleteTablet for tablet e9997e8da99c4f45a63bc3db521b33ba with delete_type TABLET_DATA_TOMBSTONED (TS 21d21a5b596b4d48a17a1fb54a972844 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33256
I20240503 08:13:10.417946 32427 tablet_bootstrap.cc:492] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap starting.
I20240503 08:13:10.420374 32471 ts_tablet_manager.cc:1888] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:13:10.420657 32471 log.cc:1196] T e9997e8da99c4f45a63bc3db521b33ba P 93f7e733cec9415eb422b0a7ba8621cb: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/wals/e9997e8da99c4f45a63bc3db521b33ba
I20240503 08:13:10.424854 31965 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.430351 32442 tablet_copy_client.cc:308] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:10.431159 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet 9b3b8b80ebd44f27899f97211266c7b2 from peer b29361eb4afa4ab3af7d8019d8a5042d ({username='slave'} at 127.20.107.136:56655)
I20240503 08:13:10.430279 30278 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.431341 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet 9b3b8b80ebd44f27899f97211266c7b2 from peer b29361eb4afa4ab3af7d8019d8a5042d at {username='slave'} at 127.20.107.136:56655: session id = b29361eb4afa4ab3af7d8019d8a5042d-9b3b8b80ebd44f27899f97211266c7b2
I20240503 08:13:10.434872 28394 tablet_copy_source_session.cc:205] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:10.436805 32442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3b8b80ebd44f27899f97211266c7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:10.439913 30671 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.454010 28218 catalog_manager.cc:4830] TS 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095): tablet e9997e8da99c4f45a63bc3db521b33ba (table rebalance_test_table_2 [id=2ff6aeab7e1847f6884750db73f8c428]) successfully deleted
W20240503 08:13:10.454465 28218 catalog_manager.cc:4810] TS 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): delete failed for tablet e9997e8da99c4f45a63bc3db521b33ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9997e8da99c4f45a63bc3db521b33ba already in progress: copying tablet
I20240503 08:13:10.458828 28227 catalog_manager.cc:5484] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d reported cstate change: config changed from index 8 to 9, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.461189 28635 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.471732 30543 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.478716 28762 consensus_queue.cc:237] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:10.487653 30675 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:10.488798 32382 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:10.494643 28635 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:10.495790 31965 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:10.496615 30675 raft_consensus.cc:1203] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:13:10.501137 28635 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.500288 31965 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.508634 28231 catalog_manager.cc:5484] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 9 to 10, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.511137 32305 tablet_bootstrap.cc:492] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:10.511651 32305 tablet_bootstrap.cc:492] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap complete.
I20240503 08:13:10.511946 32305 ts_tablet_manager.cc:1367] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844: Time spent bootstrapping tablet: real 0.179s user 0.100s sys 0.050s
I20240503 08:13:10.512907 32305 raft_consensus.cc:348] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:10.513216 32305 raft_consensus.cc:729] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Initialized, Role: LEARNER
I20240503 08:13:10.513458 32305 consensus_queue.cc:260] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:10.516337 32305 ts_tablet_manager.cc:1398] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:13:10.516966 32442 tablet_copy_client.cc:783] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:10.517048 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session 21d21a5b596b4d48a17a1fb54a972844-28e42c3244f949c7b7343bf587894ac5 received from {username='slave'} at 127.20.107.134:52703
I20240503 08:13:10.517194 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session 21d21a5b596b4d48a17a1fb54a972844-28e42c3244f949c7b7343bf587894ac5 on tablet 28e42c3244f949c7b7343bf587894ac5 with peer 21d21a5b596b4d48a17a1fb54a972844
I20240503 08:13:10.519289 32442 tablet_copy_client.cc:647] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:10.519431 30281 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:10.523689 32494 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
I20240503 08:13:10.524053 32442 tablet_copy_client.cc:523] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:10.525794 30281 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.530522 32442 tablet_bootstrap.cc:492] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap starting.
I20240503 08:13:10.536450 32374 tablet_bootstrap.cc:492] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:10.547318 30675 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.550086 32374 tablet_bootstrap.cc:492] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap complete.
I20240503 08:13:10.556591 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6e1b9f104ea471cab7527bd56433327 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:10.557107 32374 ts_tablet_manager.cc:1367] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9: Time spent bootstrapping tablet: real 0.723s user 0.048s sys 0.018s
I20240503 08:13:10.558876 32374 raft_consensus.cc:348] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:10.566233 32374 raft_consensus.cc:729] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8375219addc48fa99ac474ed1c9aef9, State: Initialized, Role: LEARNER
I20240503 08:13:10.570524 28762 consensus_queue.cc:237] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:10.570844 32374 consensus_queue.cc:260] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:10.571554 32374 ts_tablet_manager.cc:1398] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 08:13:10.572429 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session c8375219addc48fa99ac474ed1c9aef9-d44335d8be7245b1bb52b2eab9e42c18 received from {username='slave'} at 127.20.107.133:44309
I20240503 08:13:10.572558 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session c8375219addc48fa99ac474ed1c9aef9-d44335d8be7245b1bb52b2eab9e42c18 on tablet d44335d8be7245b1bb52b2eab9e42c18 with peer c8375219addc48fa99ac474ed1c9aef9
I20240503 08:13:10.575991 30523 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e1b9f104ea471cab7527bd56433327 with delete_type TABLET_DATA_TOMBSTONED (TS bea2145b8e24436392b18ec3cfe90b93 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56760
I20240503 08:13:10.579820 31976 raft_consensus.cc:1050] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 to VOTER
I20240503 08:13:10.581133 31976 consensus_queue.cc:237] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:10.582566 32328 raft_consensus.cc:1050] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d to VOTER
I20240503 08:13:10.583513 32328 consensus_queue.cc:237] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:10.584151 30675 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:10.585264 32479 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:10.586361 32304 tablet_bootstrap.cc:492] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:10.586866 32304 tablet_bootstrap.cc:492] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap complete.
I20240503 08:13:10.587204 32304 ts_tablet_manager.cc:1367] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93: Time spent bootstrapping tablet: real 0.978s user 0.061s sys 0.000s
I20240503 08:13:10.588040 32304 raft_consensus.cc:348] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:10.588318 32304 raft_consensus.cc:729] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Initialized, Role: LEARNER
I20240503 08:13:10.588550 32304 consensus_queue.cc:260] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:10.593643 31539 raft_consensus.cc:1050] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER bea2145b8e24436392b18ec3cfe90b93 to VOTER
I20240503 08:13:10.594533 31539 consensus_queue.cc:237] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:10.600217 32494 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.603971 30671 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:10.604015 30675 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.605809 32315 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:10.613936 28228 catalog_manager.cc:5484] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d reported cstate change: config changed from index 10 to 11, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.617686 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6e1b9f104ea471cab7527bd56433327 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:10.620025 28614 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e1b9f104ea471cab7527bd56433327 with delete_type TABLET_DATA_TOMBSTONED (TS 4574396f35514c9caccb346f1a7af84d not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:56340
I20240503 08:13:10.645521 32505 tablet_replica.cc:330] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: stopping tablet replica
I20240503 08:13:10.645906 32505 raft_consensus.cc:2229] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 08:13:10.646330 32505 raft_consensus.cc:2258] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 08:13:10.646772 30281 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:10.647552 32505 ts_tablet_manager.cc:1875] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:10.647576 32479 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:10.649717 30281 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.655462 32513 tablet_replica.cc:330] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: stopping tablet replica
I20240503 08:13:10.655802 32513 raft_consensus.cc:2229] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:10.656149 32513 raft_consensus.cc:2258] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:10.660849 32304 ts_tablet_manager.cc:1398] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93: Time spent starting tablet: real 0.074s user 0.003s sys 0.000s
I20240503 08:13:10.662428 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session bea2145b8e24436392b18ec3cfe90b93-2d113e8de02f447b9da0a1cb03686cd2 received from {username='slave'} at 127.20.107.135:40357
I20240503 08:13:10.662569 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-2d113e8de02f447b9da0a1cb03686cd2 on tablet 2d113e8de02f447b9da0a1cb03686cd2 with peer bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:10.666798 28762 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:10.667951 30543 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:10.669308 32315 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:10.670609 32315 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:10.675824 32513 ts_tablet_manager.cc:1875] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:10.683387 32315 raft_consensus.cc:1013] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: attempt to promote peer b29361eb4afa4ab3af7d8019d8a5042d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:13:10.684904 28635 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:10.689005 32315 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:10.717379 30413 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:10.724670 32505 ts_tablet_manager.cc:1888] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:13:10.729038 32330 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:10.735343 32513 ts_tablet_manager.cc:1888] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 08:13:10.746948 30675 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:10.749053 32511 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:10.724858 32505 log.cc:1196] T c6e1b9f104ea471cab7527bd56433327 P bea2145b8e24436392b18ec3cfe90b93: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal/wals/c6e1b9f104ea471cab7527bd56433327
I20240503 08:13:10.763926 28219 catalog_manager.cc:4830] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): tablet c6e1b9f104ea471cab7527bd56433327 (table rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]) successfully deleted
I20240503 08:13:10.747828 32330 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:10.770798 32513 log.cc:1196] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/c6e1b9f104ea471cab7527bd56433327
I20240503 08:13:10.771466 28371 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:10.772172 28217 catalog_manager.cc:4830] TS 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): tablet c6e1b9f104ea471cab7527bd56433327 (table rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]) successfully deleted
I20240503 08:13:10.772727 32330 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:10.781683 32340 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:10.782390 32340 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap complete.
I20240503 08:13:10.783036 32340 ts_tablet_manager.cc:1367] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: Time spent bootstrapping tablet: real 0.769s user 0.051s sys 0.009s
I20240503 08:13:10.784339 32340 raft_consensus.cc:348] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:10.812295 32340 raft_consensus.cc:729] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Initialized, Role: LEARNER
I20240503 08:13:10.812801 32340 consensus_queue.cc:260] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:10.813762 32340 ts_tablet_manager.cc:1398] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: Time spent starting tablet: real 0.031s user 0.003s sys 0.000s
I20240503 08:13:10.814762 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session 21d21a5b596b4d48a17a1fb54a972844-e9997e8da99c4f45a63bc3db521b33ba received from {username='slave'} at 127.20.107.134:32851
I20240503 08:13:10.814909 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session 21d21a5b596b4d48a17a1fb54a972844-e9997e8da99c4f45a63bc3db521b33ba on tablet e9997e8da99c4f45a63bc3db521b33ba with peer 21d21a5b596b4d48a17a1fb54a972844
I20240503 08:13:10.816429 30394 tablet_service.cc:1514] Processing DeleteTablet for tablet e9997e8da99c4f45a63bc3db521b33ba with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 6 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:33256
I20240503 08:13:10.816772 32492 tablet_replica.cc:330] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: stopping tablet replica
I20240503 08:13:10.817297 32492 raft_consensus.cc:2229] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 08:13:10.817642 32492 raft_consensus.cc:2258] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 08:13:10.818953 32492 ts_tablet_manager.cc:1875] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:10.825616 30412 raft_consensus.cc:1203] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:10.833709 32492 ts_tablet_manager.cc:1888] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:13:10.834106 32492 log.cc:1196] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-5/wal/wals/e9997e8da99c4f45a63bc3db521b33ba
I20240503 08:13:10.855654 28374 consensus_queue.cc:237] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:10.869247 32275 raft_consensus.cc:1050] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER bea2145b8e24436392b18ec3cfe90b93 to VOTER
I20240503 08:13:10.870071 32275 consensus_queue.cc:237] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:13:10.883042 28504 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:10.884516 28635 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:10.885972 28309 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba833ce23c4c4e81bc1a81125a1208c9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:10.886493 32086 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:10.890862 30543 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:10.891273 32395 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:10.891904 32086 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:10.896421 28504 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:10.896930 28762 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:10.897696 32400 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:10.909523 28635 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:10.910542 32086 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:10.911515 32400 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:10.926919 32395 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:10.936187 28762 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:10.944774 28635 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:10.949808 28232 catalog_manager.cc:5484] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:10.961673 28374 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:10.963366 32330 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:10.965167 28761 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:10.965924 32330 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:10.983862 30543 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:10.985687 28218 catalog_manager.cc:4830] TS 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): tablet e9997e8da99c4f45a63bc3db521b33ba (table rebalance_test_table_2 [id=2ff6aeab7e1847f6884750db73f8c428]) successfully deleted
I20240503 08:13:10.987107 32330 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:10.991780 30675 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:11.001333 28228 catalog_manager.cc:5484] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d reported cstate change: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:11.003615 30543 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
W20240503 08:13:11.013460 28438 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.131:44559, user_credentials={real_user=slave}} blocked reactor thread for 324942us
I20240503 08:13:11.015432 28631 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
W20240503 08:13:11.023694 28439 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.132:34095, user_credentials={real_user=slave}} blocked reactor thread for 317277us
I20240503 08:13:11.029022 28232 catalog_manager.cc:5484] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 5 to 6, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:11.032526 30278 raft_consensus.cc:1203] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:11.032600 32541 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:11.036294 28371 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:11.037617 28631 consensus_queue.cc:237] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:11.042037 30675 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:11.043467 31976 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:11.067070 32540 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:11.076728 30675 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:11.083297 30671 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:11.089854 28504 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:11.095016 28761 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:11.110553 30544 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:11.120213 28224 catalog_manager.cc:5484] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:11.121199 32541 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.125608 32442 tablet_bootstrap.cc:492] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:11.125159 28370 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.126116 32442 tablet_bootstrap.cc:492] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap complete.
I20240503 08:13:11.132573 28224 catalog_manager.cc:5484] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 5 to 6, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.132146 32442 ts_tablet_manager.cc:1367] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent bootstrapping tablet: real 0.602s user 0.064s sys 0.020s
I20240503 08:13:11.138016 28630 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:11.134197 32442 raft_consensus.cc:348] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:11.142149 32442 raft_consensus.cc:729] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b29361eb4afa4ab3af7d8019d8a5042d, State: Initialized, Role: LEARNER
I20240503 08:13:11.148245 28374 consensus_queue.cc:237] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:13:11.142510 32442 consensus_queue.cc:260] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:11.151515 30544 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:11.151208 28762 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.152421 32275 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:11.154363 32442 ts_tablet_manager.cc:1398] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240503 08:13:11.155118 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-9b3b8b80ebd44f27899f97211266c7b2 received from {username='slave'} at 127.20.107.136:56655
I20240503 08:13:11.155273 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-9b3b8b80ebd44f27899f97211266c7b2 on tablet 9b3b8b80ebd44f27899f97211266c7b2 with peer b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:11.156450 32400 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:11.157102 28631 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:11.159087 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:11.163954 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d13c33b8712e4c488abb99b73ff42b86 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:11.165555 32400 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:11.165555 30544 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:11.178031 28631 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:11.188439 28226 catalog_manager.cc:5484] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:11.194237 28744 tablet_service.cc:1514] Processing DeleteTablet for tablet d13c33b8712e4c488abb99b73ff42b86 with delete_type TABLET_DATA_TOMBSTONED (TS 93f7e733cec9415eb422b0a7ba8621cb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39918
I20240503 08:13:11.200999 30413 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.214156 30542 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.221320 28504 consensus_queue.cc:237] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:13:11.244177 30543 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:11.245123 32511 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:11.246812 32607 tablet_replica.cc:330] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: stopping tablet replica
I20240503 08:13:11.247093 32607 raft_consensus.cc:2229] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:11.247632 32607 raft_consensus.cc:2258] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:11.251009 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:11.265991 28373 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:11.267570 32328 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:11.270701 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:11.280934 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d60a162155504975a39c20eff88341e1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:11.288537 28226 catalog_manager.cc:5484] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:11.292102 32607 ts_tablet_manager.cc:1875] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 08:13:11.304445 28698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.133:35121, user_credentials={real_user=slave}} blocked reactor thread for 78328.9us
W20240503 08:13:11.305639 28699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.130:40397, user_credentials={real_user=slave}} blocked reactor thread for 137922us
I20240503 08:13:11.309808 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:11.318830 28371 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:11.319572 28631 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:11.320209 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:11.330051 28484 tablet_service.cc:1514] Processing DeleteTablet for tablet d60a162155504975a39c20eff88341e1 with delete_type TABLET_DATA_TOMBSTONED (TS 716ffbb2a00b4f43b03e5cedcc9d302a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54162
I20240503 08:13:11.334823 32511 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20240503 08:13:11.336418 30523 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:11.345744 28483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:11.347962 32617 tablet_replica.cc:330] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: stopping tablet replica
I20240503 08:13:11.348317 32617 raft_consensus.cc:2229] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:11.358582 32617 raft_consensus.cc:2258] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:11.363075 32617 ts_tablet_manager.cc:1875] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:11.375870 30675 raft_consensus.cc:1203] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:11.383304 32622 ts_tablet_manager.cc:902] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:11.387959 32541 raft_consensus.cc:1050] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER bea2145b8e24436392b18ec3cfe90b93 to VOTER
I20240503 08:13:11.389528 32541 consensus_queue.cc:237] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:11.390880 28761 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:11.391592 32572 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:11.391966 28635 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:11.395696 32617 ts_tablet_manager.cc:1888] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:13:11.395853 32617 log.cc:1196] T d60a162155504975a39c20eff88341e1 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/d60a162155504975a39c20eff88341e1
I20240503 08:13:11.396728 28218 catalog_manager.cc:4830] TS 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): tablet d60a162155504975a39c20eff88341e1 (table rebalance_test_table_3 [id=eeb192513dc04f0eb0552ff611d314c0]) successfully deleted
I20240503 08:13:11.397061 32607 ts_tablet_manager.cc:1888] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:11.397208 32607 log.cc:1196] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/wals/d13c33b8712e4c488abb99b73ff42b86
I20240503 08:13:11.398149 28218 catalog_manager.cc:4830] TS 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095): tablet d13c33b8712e4c488abb99b73ff42b86 (table rebalance_test_table_2 [id=2ff6aeab7e1847f6884750db73f8c428]) successfully deleted
I20240503 08:13:11.406688 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b5736ee8065344ea8124fdf759035216 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:11.407620 32527 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:11.408202 30543 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:11.415076 28762 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:11.420150 28228 catalog_manager.cc:5484] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93 reported cstate change: config changed from index 6 to 7, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:11.429764 32328 raft_consensus.cc:1050] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 to VOTER
I20240503 08:13:11.430588 32328 consensus_queue.cc:237] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:11.432312 32377 tablet_bootstrap.cc:492] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:11.438493 28373 consensus_queue.cc:237] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
W20240503 08:13:11.441926 28217 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 2d113e8de02f447b9da0a1cb03686cd2 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:11.443629 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:11.446902 30413 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:11.447520 32532 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:11.453698 28762 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:11.454210 32532 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:11.456776 32330 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.457572 28762 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.462011 28371 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:11.462706 32532 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:11.462462 30413 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.464357 28374 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.471076 30543 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:11.471992 32541 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:11.472769 28227 catalog_manager.cc:5484] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 6 to 7, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.474381 30413 raft_consensus.cc:1261] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:11.475034 28761 raft_consensus.cc:1261] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:11.475287 32328 consensus_queue.cc:1035] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:11.475739 32532 consensus_queue.cc:1035] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:11.484506 28371 raft_consensus.cc:1261] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:11.486163 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:11.488554 28217 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 2d113e8de02f447b9da0a1cb03686cd2 with cas_config_opid_index 6: aborting the task: latest config opid_index 7; task opid_index 6
I20240503 08:13:11.492724 32541 consensus_queue.cc:1035] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:11.509018 32377 tablet_bootstrap.cc:492] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9: Bootstrap complete.
I20240503 08:13:11.509534 32377 ts_tablet_manager.cc:1367] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9: Time spent bootstrapping tablet: real 1.460s user 0.053s sys 0.010s
I20240503 08:13:11.516831 28631 consensus_queue.cc:237] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:11.517360 32377 raft_consensus.cc:348] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:11.517766 32377 raft_consensus.cc:729] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8375219addc48fa99ac474ed1c9aef9, State: Initialized, Role: LEARNER
I20240503 08:13:11.518071 32377 consensus_queue.cc:260] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:11.519817 32377 ts_tablet_manager.cc:1398] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 08:13:11.519991 30413 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:11.520838 28763 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:11.521214 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session c8375219addc48fa99ac474ed1c9aef9-a8a9adc3712d436c95505a3216e7607c received from {username='slave'} at 127.20.107.133:49403
I20240503 08:13:11.521395 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session c8375219addc48fa99ac474ed1c9aef9-a8a9adc3712d436c95505a3216e7607c on tablet a8a9adc3712d436c95505a3216e7607c with peer c8375219addc48fa99ac474ed1c9aef9
I20240503 08:13:11.521358 30543 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.523551 32328 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240503 08:13:11.524741 32328 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:11.528589 32622 tablet_copy_client.cc:308] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:11.529332 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet ba833ce23c4c4e81bc1a81125a1208c9 from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:40941)
I20240503 08:13:11.529518 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet ba833ce23c4c4e81bc1a81125a1208c9 from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:40941: session id = bea2145b8e24436392b18ec3cfe90b93-ba833ce23c4c4e81bc1a81125a1208c9
I20240503 08:13:11.530572 32532 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.532496 28394 tablet_copy_source_session.cc:205] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:11.536402 30412 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.541672 28374 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.543707 28227 catalog_manager.cc:5484] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.546463 32622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba833ce23c4c4e81bc1a81125a1208c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:11.548771 31976 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.549726 28762 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.552896 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d113e8de02f447b9da0a1cb03686cd2 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:11.554064 28761 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.555307 30413 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.559263 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:11.579005 32427 tablet_bootstrap.cc:492] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:11.584203 32427 tablet_bootstrap.cc:492] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap complete.
I20240503 08:13:11.584628 32427 ts_tablet_manager.cc:1367] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9: Time spent bootstrapping tablet: real 1.167s user 0.054s sys 0.006s
I20240503 08:13:11.585690 32427 raft_consensus.cc:348] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:11.586134 32427 raft_consensus.cc:729] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8375219addc48fa99ac474ed1c9aef9, State: Initialized, Role: LEARNER
I20240503 08:13:11.586397 32427 consensus_queue.cc:260] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:11.588100 32427 ts_tablet_manager.cc:1398] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:13:11.589037 28524 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session c8375219addc48fa99ac474ed1c9aef9-a8b4e398d34443b4a440f9ba48555d84 received from {username='slave'} at 127.20.107.133:43991
I20240503 08:13:11.589159 28524 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session c8375219addc48fa99ac474ed1c9aef9-a8b4e398d34443b4a440f9ba48555d84 on tablet a8b4e398d34443b4a440f9ba48555d84 with peer c8375219addc48fa99ac474ed1c9aef9
I20240503 08:13:11.591578 30656 tablet_service.cc:1514] Processing DeleteTablet for tablet b5736ee8065344ea8124fdf759035216 with delete_type TABLET_DATA_TOMBSTONED (TS b29361eb4afa4ab3af7d8019d8a5042d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53250
I20240503 08:13:11.595136 28232 catalog_manager.cc:5484] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 7 to 8, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:11.600124 30671 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:11.600689 28763 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:11.602721 32400 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:11.603228 32400 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:13:11.604333 28308 consensus_peers.cc:485] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b3b8b80ebd44f27899f97211266c7b2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:11.606475 28504 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:11.607165 32559 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:11.607468 32328 raft_consensus.cc:1050] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER c8375219addc48fa99ac474ed1c9aef9 to VOTER
I20240503 08:13:11.608341 32328 consensus_queue.cc:237] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:11.610970 30542 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:11.612129 32532 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:11.616670 30543 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.625301 30278 raft_consensus.cc:1203] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:11.629030 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:11.629161 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:11.639508 32642 tablet_replica.cc:330] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d: stopping tablet replica
I20240503 08:13:11.640431 32642 raft_consensus.cc:2229] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Raft consensus shutting down.
I20240503 08:13:11.641889 32642 raft_consensus.cc:2258] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Raft consensus is shut down!
I20240503 08:13:11.642131 30278 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:11.642875 32540 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:11.649345 32400 raft_consensus.cc:1013] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: attempt to promote peer b29361eb4afa4ab3af7d8019d8a5042d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:13:11.658475 28373 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:11.659190 32540 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:11.659869 28502 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:11.660589 32540 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:11.661381 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:11.672119 32642 ts_tablet_manager.cc:1875] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:11.684455 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:11.693549 31976 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:11.694037 28353 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d113e8de02f447b9da0a1cb03686cd2 with delete_type TABLET_DATA_TOMBSTONED (TS 4e4a1242f3714d03a6c9ee51dd63929c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:51022
I20240503 08:13:11.696240 28631 consensus_queue.cc:237] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:11.698920 30412 raft_consensus.cc:1261] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:11.698881 28630 consensus_queue.cc:237] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:11.699146 28374 raft_consensus.cc:1261] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:11.698925 28373 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:11.699676 30278 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:11.701493 32330 consensus_queue.cc:1035] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:11.702623 32532 consensus_queue.cc:1035] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:11.702699 30412 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:11.704424 32540 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:11.705372 32330 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.706094 30412 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.707334 28374 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.710006 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28e42c3244f949c7b7343bf587894ac5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:11.713753 32532 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.713963 28226 catalog_manager.cc:5484] T 28e42c3244f949c7b7343bf587894ac5 P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.715425 30412 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.718029 28744 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e42c3244f949c7b7343bf587894ac5 with delete_type TABLET_DATA_TOMBSTONED (TS 93f7e733cec9415eb422b0a7ba8621cb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39918
I20240503 08:13:11.718376 32607 tablet_replica.cc:330] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb: stopping tablet replica
I20240503 08:13:11.718546 32607 raft_consensus.cc:2229] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:11.718744 32607 raft_consensus.cc:2258] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:11.719175 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d113e8de02f447b9da0a1cb03686cd2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:11.719405 28226 catalog_manager.cc:5484] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 reported cstate change: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:11.721489 32607 ts_tablet_manager.cc:1875] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:11.727149 28761 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:11.730717 32328 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:11.731572 32275 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:11.732283 28763 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.737146 32642 ts_tablet_manager.cc:1888] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:13:11.734998 32652 tablet_replica.cc:330] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c: stopping tablet replica
I20240503 08:13:11.737723 32652 raft_consensus.cc:2229] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:11.737406 28765 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:11.738037 32642 log.cc:1196] T b5736ee8065344ea8124fdf759035216 P b29361eb4afa4ab3af7d8019d8a5042d: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal/wals/b5736ee8065344ea8124fdf759035216
I20240503 08:13:11.738960 28218 catalog_manager.cc:4830] TS b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): tablet b5736ee8065344ea8124fdf759035216 (table rebalance_test_table_7 [id=0225804a5c4a42cbb32c4b0029752bd1]) successfully deleted
I20240503 08:13:11.740685 32607 ts_tablet_manager.cc:1888] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:11.740857 32607 log.cc:1196] T 28e42c3244f949c7b7343bf587894ac5 P 93f7e733cec9415eb422b0a7ba8621cb: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/wals/28e42c3244f949c7b7343bf587894ac5
I20240503 08:13:11.741794 32652 raft_consensus.cc:2258] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:11.742048 28218 catalog_manager.cc:4830] TS 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095): tablet 28e42c3244f949c7b7343bf587894ac5 (table rebalance_test_table_3 [id=eeb192513dc04f0eb0552ff611d314c0]) successfully deleted
I20240503 08:13:11.742571 28227 catalog_manager.cc:5484] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: config changed from index 8 to 9, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:11.743956 30675 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:11.747588 30523 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:11.743885 32652 ts_tablet_manager.cc:1875] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:11.753244 30523 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d113e8de02f447b9da0a1cb03686cd2 with delete_type TABLET_DATA_TOMBSTONED (TS bea2145b8e24436392b18ec3cfe90b93 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56760
I20240503 08:13:11.757222 28227 catalog_manager.cc:5484] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d reported cstate change: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:11.765507 28502 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:11.769300 30278 raft_consensus.cc:1203] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:11.778836 32654 tablet_replica.cc:330] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93: stopping tablet replica
I20240503 08:13:11.779475 32654 raft_consensus.cc:2229] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:11.779863 32654 raft_consensus.cc:2258] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:11.783810 32559 raft_consensus.cc:1050] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d to VOTER
I20240503 08:13:11.784778 32559 consensus_queue.cc:237] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:11.784960 28505 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
W20240503 08:13:11.789499 28308 consensus_peers.cc:485] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b3b8b80ebd44f27899f97211266c7b2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:11.801935 28502 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:11.802645 32400 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:11.805359 28761 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:11.807271 32660 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:11.812212 28761 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:11.813235 30671 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:11.813817 32275 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:11.815397 30675 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:11.822872 32622 tablet_copy_client.cc:783] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:11.822352 28227 catalog_manager.cc:5484] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d reported cstate change: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:11.823148 32622 tablet_copy_client.cc:647] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:11.828732 32652 ts_tablet_manager.cc:1888] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:13:11.828907 32652 log.cc:1196] T 2d113e8de02f447b9da0a1cb03686cd2 P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/2d113e8de02f447b9da0a1cb03686cd2
I20240503 08:13:11.830109 28218 catalog_manager.cc:4830] TS 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): tablet 2d113e8de02f447b9da0a1cb03686cd2 (table rebalance_test_table_8 [id=649380be96b24c78aa15b40471703d93]) successfully deleted
I20240503 08:13:11.837527 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:11.841742 32655 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:11.847013 32654 ts_tablet_manager.cc:1875] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:11.872537 32622 tablet_copy_client.cc:523] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:11.936945 28502 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:11.946023 28502 consensus_queue.cc:237] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:11.953956 28373 consensus_queue.cc:237] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:11.956535 28761 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:11.956576 30671 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:11.957641 32655 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:13:11.957986 28308 consensus_peers.cc:485] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b3b8b80ebd44f27899f97211266c7b2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:11.958235 32400 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:11.962579 28630 raft_consensus.cc:1261] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:11.963543 32572 consensus_queue.cc:1035] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:11.964113 32400 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:11.966686 28761 raft_consensus.cc:1261] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:11.967363 32511 consensus_queue.cc:1035] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:11.972630 30675 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
W20240503 08:13:11.974318 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da529c8f8de241d38678d1596b9402ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:11.974725 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b3b8b80ebd44f27899f97211266c7b2 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:11.974860 28761 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:11.981442 28226 catalog_manager.cc:5484] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d reported cstate change: config changed from index 6 to 7, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:11.988556 28484 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b3b8b80ebd44f27899f97211266c7b2 with delete_type TABLET_DATA_TOMBSTONED (TS 716ffbb2a00b4f43b03e5cedcc9d302a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54162
I20240503 08:13:11.995085 32679 tablet_replica.cc:330] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a: stopping tablet replica
I20240503 08:13:11.995395 32679 raft_consensus.cc:2229] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:11.995723 32679 raft_consensus.cc:2258] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:12.001713 32679 ts_tablet_manager.cc:1875] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:12.004809 32622 tablet_bootstrap.cc:492] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
I20240503 08:13:12.019925 32275 raft_consensus.cc:1050] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER c8375219addc48fa99ac474ed1c9aef9 to VOTER
I20240503 08:13:12.020761 32275 consensus_queue.cc:237] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:12.031147 32654 ts_tablet_manager.cc:1888] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:12.031337 32654 log.cc:1196] T 2d113e8de02f447b9da0a1cb03686cd2 P bea2145b8e24436392b18ec3cfe90b93: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal/wals/2d113e8de02f447b9da0a1cb03686cd2
I20240503 08:13:12.040185 32679 ts_tablet_manager.cc:1888] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:13:12.040422 28219 catalog_manager.cc:4830] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): tablet 2d113e8de02f447b9da0a1cb03686cd2 (table rebalance_test_table_8 [id=649380be96b24c78aa15b40471703d93]) successfully deleted
I20240503 08:13:12.043350 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:12.047327 32679 log.cc:1196] T 9b3b8b80ebd44f27899f97211266c7b2 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/9b3b8b80ebd44f27899f97211266c7b2
I20240503 08:13:12.048241 28218 catalog_manager.cc:4830] TS 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): tablet 9b3b8b80ebd44f27899f97211266c7b2 (table rebalance_test_table_6 [id=87e8ca2152ac403cb71e5eb67c0b2e0e]) successfully deleted
I20240503 08:13:12.058084 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:12.058421 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:12.086660 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:12.092898 32681 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:12.106350 28762 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:12.112360 28502 raft_consensus.cc:1261] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:12.112627 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:12.113101 32655 consensus_queue.cc:1035] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:12.114197 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:12.118424 30278 raft_consensus.cc:1261] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:12.119123 32655 consensus_queue.cc:1035] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:12.126461 28761 raft_consensus.cc:1261] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:12.127272 32655 consensus_queue.cc:1035] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:12.142740 28631 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:12.143672 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:12.144587 28505 consensus_queue.cc:237] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:13:12.147001 30542 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:12.154179 28227 catalog_manager.cc:5484] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index -1 to 4, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:12.167057 28762 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:12.168694 32655 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:12.173734 30523 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:12.178366 28761 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:12.181236 30278 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:12.194496 28232 catalog_manager.cc:5484] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:12.196002 28505 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:12.229740 28373 consensus_queue.cc:237] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:12.233083 28761 raft_consensus.cc:1261] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:12.234071 30278 raft_consensus.cc:1261] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:12.153041 32511 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.082s
I20240503 08:13:12.234802 32661 consensus_queue.cc:1035] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 08:13:12.235088 32682 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:12.236474 32275 consensus_queue.cc:1035] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:12.237533 32685 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:12.238835 30542 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:12.239693 32661 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:12.241916 28762 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:12.245034 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a8a9adc3712d436c95505a3216e7607c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:12.246330 28761 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:12.246711 30281 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:12.248378 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b5736ee8065344ea8124fdf759035216 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:12.250356 28226 catalog_manager.cc:5484] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 5 to 6, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:12.254423 28484 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a9adc3712d436c95505a3216e7607c with delete_type TABLET_DATA_TOMBSTONED (TS 716ffbb2a00b4f43b03e5cedcc9d302a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54162
I20240503 08:13:12.254791 32679 tablet_replica.cc:330] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: stopping tablet replica
I20240503 08:13:12.255003 32679 raft_consensus.cc:2229] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:12.255234 32679 raft_consensus.cc:2258] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:12.256417 32679 ts_tablet_manager.cc:1875] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:12.256435 28227 catalog_manager.cc:5484] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93 reported cstate change: config changed from index 7 to 8, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:12.264376 32679 ts_tablet_manager.cc:1888] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:12.270282 28614 tablet_service.cc:1514] Processing DeleteTablet for tablet b5736ee8065344ea8124fdf759035216 with delete_type TABLET_DATA_TOMBSTONED (TS 4574396f35514c9caccb346f1a7af84d not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:56340
I20240503 08:13:12.283907 32679 log.cc:1196] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/a8a9adc3712d436c95505a3216e7607c
I20240503 08:13:12.285054 28218 catalog_manager.cc:4830] TS 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): tablet a8a9adc3712d436c95505a3216e7607c (table rebalance_test_table_8 [id=649380be96b24c78aa15b40471703d93]) successfully deleted
I20240503 08:13:12.288530 32700 tablet_replica.cc:330] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: stopping tablet replica
I20240503 08:13:12.288928 32700 raft_consensus.cc:2229] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:12.289541 32700 raft_consensus.cc:2258] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:12.291750 32700 ts_tablet_manager.cc:1875] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:12.308202 28761 consensus_queue.cc:237] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
W20240503 08:13:12.326519 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 564fb1b51abd4f8da0372ec99e3c599c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:12.327538 28630 raft_consensus.cc:1261] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:12.328680 32479 consensus_queue.cc:1035] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:12.332504 32700 ts_tablet_manager.cc:1888] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:13:12.332692 32700 log.cc:1196] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/b5736ee8065344ea8124fdf759035216
I20240503 08:13:12.334996 28373 raft_consensus.cc:1261] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:12.338014 32479 consensus_queue.cc:1035] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:12.345772 32511 raft_consensus.cc:1050] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER c8375219addc48fa99ac474ed1c9aef9 to VOTER
I20240503 08:13:12.346915 32511 consensus_queue.cc:237] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:12.350028 28217 catalog_manager.cc:4830] TS 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): tablet b5736ee8065344ea8124fdf759035216 (table rebalance_test_table_7 [id=0225804a5c4a42cbb32c4b0029752bd1]) successfully deleted
I20240503 08:13:12.356280 30281 raft_consensus.cc:1261] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:12.357170 32685 consensus_queue.cc:1035] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:12.362643 28761 raft_consensus.cc:1261] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:12.363500 32685 consensus_queue.cc:1035] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:12.366161 28630 raft_consensus.cc:1261] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:12.366961 32685 consensus_queue.cc:1035] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:12.400383 32684 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:12.401549 28630 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:12.403177 32479 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
W20240503 08:13:12.407212 28699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.134:34587, user_credentials={real_user=slave}} blocked reactor thread for 70992.2us
I20240503 08:13:12.407809 30278 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:12.412894 28232 catalog_manager.cc:5484] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:12.415506 28761 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:12.424199 28227 catalog_manager.cc:5484] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:12.427717 28505 consensus_queue.cc:237] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:12.430347 30278 raft_consensus.cc:1261] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:12.431535 32682 consensus_queue.cc:1035] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:12.437338 28761 raft_consensus.cc:1261] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:12.437992 32684 consensus_queue.cc:1035] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:12.440670 32511 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:12.445325 32622 tablet_bootstrap.cc:492] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:12.446228 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a8b4e398d34443b4a440f9ba48555d84 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:12.449194 32622 tablet_bootstrap.cc:492] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap complete.
I20240503 08:13:12.449414 28227 catalog_manager.cc:5484] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:12.450214 32622 ts_tablet_manager.cc:1367] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93: Time spent bootstrapping tablet: real 0.517s user 0.047s sys 0.011s
I20240503 08:13:12.452468 28761 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:12.454102 28614 tablet_service.cc:1514] Processing DeleteTablet for tablet a8b4e398d34443b4a440f9ba48555d84 with delete_type TABLET_DATA_TOMBSTONED (TS 4574396f35514c9caccb346f1a7af84d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56340
I20240503 08:13:12.454455 32700 tablet_replica.cc:330] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d: stopping tablet replica
I20240503 08:13:12.454654 32700 raft_consensus.cc:2229] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:12.454877 32700 raft_consensus.cc:2258] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:12.455291 32722 ts_tablet_manager.cc:902] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:12.456126 32700 ts_tablet_manager.cc:1875] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:12.461009 28630 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:12.464324 32700 ts_tablet_manager.cc:1888] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:12.465461 32700 log.cc:1196] T a8b4e398d34443b4a440f9ba48555d84 P 4574396f35514c9caccb346f1a7af84d: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/a8b4e398d34443b4a440f9ba48555d84
I20240503 08:13:12.466950 28217 catalog_manager.cc:4830] TS 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): tablet a8b4e398d34443b4a440f9ba48555d84 (table rebalance_test_table_3 [id=eeb192513dc04f0eb0552ff611d314c0]) successfully deleted
I20240503 08:13:12.467176 32622 raft_consensus.cc:348] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:12.467486 32622 raft_consensus.cc:729] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Initialized, Role: LEARNER
I20240503 08:13:12.467722 32622 consensus_queue.cc:260] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:12.468484 32622 ts_tablet_manager.cc:1398] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93: Time spent starting tablet: real 0.018s user 0.000s sys 0.000s
I20240503 08:13:12.471980 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session bea2145b8e24436392b18ec3cfe90b93-ba833ce23c4c4e81bc1a81125a1208c9 received from {username='slave'} at 127.20.107.135:40941
I20240503 08:13:12.472154 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-ba833ce23c4c4e81bc1a81125a1208c9 on tablet ba833ce23c4c4e81bc1a81125a1208c9 with peer bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:12.473393 30278 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:12.493314 28373 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:12.534322 28761 consensus_queue.cc:237] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:13:12.538627 30412 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:13:12.538837 28698 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): Couldn't send request to peer 4574396f35514c9caccb346f1a7af84d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:12.539336 32709 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:12.542879 28505 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:12.544400 32709 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:12.546744 32722 tablet_copy_client.cc:308] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:12.549360 28630 consensus_queue.cc:237] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:12.561200 32709 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:12.562623 32745 ts_tablet_manager.cc:902] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:13:12.564561 28373 raft_consensus.cc:1261] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:12.565611 30278 raft_consensus.cc:1261] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:12.567016 32328 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:13:12.567528 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 585ba864cc384e7f9db90e9d4c47e82c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:12.567771 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet 9b3b8b80ebd44f27899f97211266c7b2 from peer 21d21a5b596b4d48a17a1fb54a972844 ({username='slave'} at 127.20.107.134:32851)
I20240503 08:13:12.567924 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet 9b3b8b80ebd44f27899f97211266c7b2 from peer 21d21a5b596b4d48a17a1fb54a972844 at {username='slave'} at 127.20.107.134:32851: session id = 21d21a5b596b4d48a17a1fb54a972844-9b3b8b80ebd44f27899f97211266c7b2
I20240503 08:13:12.572469 32328 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:12.572710 28394 tablet_copy_source_session.cc:205] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:12.573632 30412 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:12.574158 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3b8b80ebd44f27899f97211266c7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:12.583987 28227 catalog_manager.cc:5484] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 6 to 7, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:12.585268 28502 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:12.596487 28373 consensus_queue.cc:237] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:12.598213 32541 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:12.600021 28371 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:12.607343 32722 tablet_copy_client.cc:783] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 0 data blocks...
W20240503 08:13:12.609894 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 173376257c4946b5a801a27e4dffe90d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:12.611120 28505 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:12.611734 32559 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:12.615489 28228 catalog_manager.cc:5484] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 6 to 7, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:12.620519 30281 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:12.622643 28630 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:12.623319 32559 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:12.641481 32559 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:12.647866 28505 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:12.654749 28232 catalog_manager.cc:5484] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:12.662109 28631 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:12.673408 32722 tablet_copy_client.cc:647] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:12.677378 32722 tablet_copy_client.cc:523] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:12.682479 32722 tablet_bootstrap.cc:492] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap starting.
I20240503 08:13:12.694476 28761 consensus_queue.cc:237] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
W20240503 08:13:12.700054 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0caf86850b3841ea811b0a6ab19ae330. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:12.700716 30412 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:12.700968 28502 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:12.701660 32709 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:12.702239 32709 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:12.739099 28373 consensus_queue.cc:237] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:12.758679 32716 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:12.761718 28505 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:12.770676 28228 catalog_manager.cc:5484] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:13:12.780998 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:12.783329 30675 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:12.784073 32559 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:12.791229 28502 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:12.792055 32661 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:12.796192 30412 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:12.805739 32745 tablet_copy_client.cc:308] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:13:12.806914 28524 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet da529c8f8de241d38678d1596b9402ac from peer b29361eb4afa4ab3af7d8019d8a5042d ({username='slave'} at 127.20.107.136:33075)
I20240503 08:13:12.807109 28524 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet da529c8f8de241d38678d1596b9402ac from peer b29361eb4afa4ab3af7d8019d8a5042d at {username='slave'} at 127.20.107.136:33075: session id = b29361eb4afa4ab3af7d8019d8a5042d-da529c8f8de241d38678d1596b9402ac
I20240503 08:13:12.811856 28524 tablet_copy_source_session.cc:205] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:12.814016 32745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da529c8f8de241d38678d1596b9402ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:12.819833 32559 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:12.820890 326 ts_tablet_manager.cc:902] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:13:12.830165 28232 catalog_manager.cc:5484] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 8 to 9, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:12.855360 28630 consensus_queue.cc:237] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:12.858953 28761 consensus_queue.cc:237] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
W20240503 08:13:12.862075 28698 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): Couldn't send request to peer 4574396f35514c9caccb346f1a7af84d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:12.865458 28501 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:12.867586 32709 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:13:12.868366 28700 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4babf0ea9b954a23b1b1c4e403dce0bd. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:12.871011 32716 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:12.871702 28501 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:12.877974 32745 tablet_copy_client.cc:783] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:12.878322 32745 tablet_copy_client.cc:647] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:12.879038 30542 raft_consensus.cc:1203] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:12.879262 30412 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:12.879943 32709 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:12.880859 28224 catalog_manager.cc:5484] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:12.881611 30414 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:12.840906 28502 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:12.900059 32745 tablet_copy_client.cc:523] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:12.903306 30542 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:13:12.904138 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4ceeeb3495e4a8fac6e13e431c64af4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:12.904417 32328 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:12.912101 28373 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:12.913003 32328 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:12.915241 32330 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:12.921952 30543 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:12.930270 28227 catalog_manager.cc:5484] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 reported cstate change: config changed from index 6 to 7, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:12.932420 28502 consensus_queue.cc:237] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:12.933007 30675 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:12.934069 326 tablet_copy_client.cc:308] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:13:12.936220 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 564fb1b51abd4f8da0372ec99e3c599c from peer c8375219addc48fa99ac474ed1c9aef9 ({username='slave'} at 127.20.107.133:41339)
I20240503 08:13:12.936405 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 564fb1b51abd4f8da0372ec99e3c599c from peer c8375219addc48fa99ac474ed1c9aef9 at {username='slave'} at 127.20.107.133:41339: session id = c8375219addc48fa99ac474ed1c9aef9-564fb1b51abd4f8da0372ec99e3c599c
I20240503 08:13:12.939687 28785 tablet_copy_source_session.cc:205] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:12.941252 28374 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:12.942117 326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564fb1b51abd4f8da0372ec99e3c599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:12.975546 32745 tablet_bootstrap.cc:492] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap starting.
I20240503 08:13:13.020820 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 08:13:13.046520 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a20489bf65774902af38d228ee966ffe. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.048032 28373 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:13.048640 32722 tablet_bootstrap.cc:492] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:13.048609 32682 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:13.049130 32722 tablet_bootstrap.cc:492] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap complete.
I20240503 08:13:13.049458 32722 ts_tablet_manager.cc:1367] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844: Time spent bootstrapping tablet: real 0.367s user 0.060s sys 0.008s
I20240503 08:13:13.050361 32722 raft_consensus.cc:348] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:13.050774 32722 raft_consensus.cc:729] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Initialized, Role: LEARNER
I20240503 08:13:13.051055 32722 consensus_queue.cc:260] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:13.053027 30543 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:13.056898 32722 ts_tablet_manager.cc:1398] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:13:13.061599 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session 21d21a5b596b4d48a17a1fb54a972844-9b3b8b80ebd44f27899f97211266c7b2 received from {username='slave'} at 127.20.107.134:32851
I20240503 08:13:13.061781 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session 21d21a5b596b4d48a17a1fb54a972844-9b3b8b80ebd44f27899f97211266c7b2 on tablet 9b3b8b80ebd44f27899f97211266c7b2 with peer 21d21a5b596b4d48a17a1fb54a972844
I20240503 08:13:13.078058 32682 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:13.079866 369 ts_tablet_manager.cc:902] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:13:13.083171 28374 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:13.085680 32722 ts_tablet_manager.cc:902] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:13.090430 32682 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:13.093874 326 tablet_copy_client.cc:783] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.094257 326 tablet_copy_client.cc:647] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.109683 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:13.124650 326 tablet_copy_client.cc:523] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.142560 369 tablet_copy_client.cc:308] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:13:13.143406 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 0caf86850b3841ea811b0a6ab19ae330 from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:43079)
I20240503 08:13:13.143620 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 0caf86850b3841ea811b0a6ab19ae330 from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:43079: session id = bea2145b8e24436392b18ec3cfe90b93-0caf86850b3841ea811b0a6ab19ae330
I20240503 08:13:13.146797 28785 tablet_copy_source_session.cc:205] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.157447 385 ts_tablet_manager.cc:902] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:13:13.158097 28228 catalog_manager.cc:5484] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 6 to 7, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:13.165779 28761 consensus_queue.cc:237] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
W20240503 08:13:13.168748 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): Couldn't send request to peer 4574396f35514c9caccb346f1a7af84d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.169826 30414 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:13.170404 32716 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:13.171504 32745 tablet_bootstrap.cc:492] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:13.171993 32745 tablet_bootstrap.cc:492] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap complete.
I20240503 08:13:13.172302 32745 ts_tablet_manager.cc:1367] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d: Time spent bootstrapping tablet: real 0.197s user 0.056s sys 0.007s
I20240503 08:13:13.174041 28502 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:13.175598 32718 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:13.176257 369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caf86850b3841ea811b0a6ab19ae330. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.179111 30412 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:13.179600 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:13.182721 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:13.185267 30414 raft_consensus.cc:1203] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:13.186910 28502 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:13.176543 32718 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
W20240503 08:13:13.196588 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0caf86850b3841ea811b0a6ab19ae330. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.196997 385 tablet_copy_client.cc:308] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:13:13.197894 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet 585ba864cc384e7f9db90e9d4c47e82c from peer 21d21a5b596b4d48a17a1fb54a972844 ({username='slave'} at 127.20.107.134:52703)
I20240503 08:13:13.198076 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet 585ba864cc384e7f9db90e9d4c47e82c from peer 21d21a5b596b4d48a17a1fb54a972844 at {username='slave'} at 127.20.107.134:52703: session id = 21d21a5b596b4d48a17a1fb54a972844-585ba864cc384e7f9db90e9d4c47e82c
I20240503 08:13:13.199856 30543 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:13.202824 28655 tablet_copy_source_session.cc:205] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.205101 385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585ba864cc384e7f9db90e9d4c47e82c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.205904 28227 catalog_manager.cc:5484] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: config changed from index 7 to 8, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:13.206542 369 tablet_copy_client.cc:783] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.206815 369 tablet_copy_client.cc:647] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.213410 369 tablet_copy_client.cc:523] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.216274 385 tablet_copy_client.cc:783] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.219848 385 tablet_copy_client.cc:647] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.220937 32745 raft_consensus.cc:348] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:13.221222 32745 raft_consensus.cc:729] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b29361eb4afa4ab3af7d8019d8a5042d, State: Initialized, Role: LEARNER
I20240503 08:13:13.222591 385 tablet_copy_client.cc:523] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.224898 32745 consensus_queue.cc:260] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:13.225888 32745 ts_tablet_manager.cc:1398] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
I20240503 08:13:13.230811 385 tablet_bootstrap.cc:492] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap starting.
I20240503 08:13:13.233378 28524 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-da529c8f8de241d38678d1596b9402ac received from {username='slave'} at 127.20.107.136:33075
I20240503 08:13:13.233556 28524 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-da529c8f8de241d38678d1596b9402ac on tablet da529c8f8de241d38678d1596b9402ac with peer b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:13.171159 326 tablet_bootstrap.cc:492] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9: Bootstrap starting.
I20240503 08:13:13.249428 369 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
I20240503 08:13:13.259079 32722 tablet_copy_client.cc:308] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:13.260022 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet 173376257c4946b5a801a27e4dffe90d from peer 21d21a5b596b4d48a17a1fb54a972844 ({username='slave'} at 127.20.107.134:32851)
I20240503 08:13:13.260217 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet 173376257c4946b5a801a27e4dffe90d from peer 21d21a5b596b4d48a17a1fb54a972844 at {username='slave'} at 127.20.107.134:32851: session id = 21d21a5b596b4d48a17a1fb54a972844-173376257c4946b5a801a27e4dffe90d
I20240503 08:13:13.263974 28394 tablet_copy_source_session.cc:205] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.266140 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 173376257c4946b5a801a27e4dffe90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.273631 30523 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:13.300312 32722 tablet_copy_client.cc:783] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.300585 32722 tablet_copy_client.cc:647] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.303644 32722 tablet_copy_client.cc:523] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.315937 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:13.316119 32722 tablet_bootstrap.cc:492] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap starting.
I20240503 08:13:13.329887 28502 consensus_queue.cc:237] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:13.333597 28374 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:13:13.335940 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a20489bf65774902af38d228ee966ffe. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.336082 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a20489bf65774902af38d228ee966ffe. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.336318 32684 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:13.338966 30542 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:13.339812 32682 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:13.347143 28374 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:13.350474 32511 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:13.353271 30542 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:13.367476 385 tablet_bootstrap.cc:492] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:13.368712 28227 catalog_manager.cc:5484] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 7 to 8, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:13.393193 385 tablet_bootstrap.cc:492] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap complete.
I20240503 08:13:13.393576 385 ts_tablet_manager.cc:1367] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844: Time spent bootstrapping tablet: real 0.163s user 0.064s sys 0.001s
I20240503 08:13:13.394449 385 raft_consensus.cc:348] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:13.394734 385 raft_consensus.cc:729] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Initialized, Role: LEARNER
I20240503 08:13:13.394963 385 consensus_queue.cc:260] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:13.395571 385 ts_tablet_manager.cc:1398] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:13.398000 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session 21d21a5b596b4d48a17a1fb54a972844-585ba864cc384e7f9db90e9d4c47e82c received from {username='slave'} at 127.20.107.134:52703
I20240503 08:13:13.398155 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session 21d21a5b596b4d48a17a1fb54a972844-585ba864cc384e7f9db90e9d4c47e82c on tablet 585ba864cc384e7f9db90e9d4c47e82c with peer 21d21a5b596b4d48a17a1fb54a972844
I20240503 08:13:13.400171 385 ts_tablet_manager.cc:902] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:13.426806 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:13.432155 417 ts_tablet_manager.cc:902] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:13:13.434813 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:13.443586 369 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:13.457660 32745 ts_tablet_manager.cc:902] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:13:13.466717 433 ts_tablet_manager.cc:902] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:13:13.472520 326 tablet_bootstrap.cc:492] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:13.473091 326 tablet_bootstrap.cc:492] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9: Bootstrap complete.
I20240503 08:13:13.473397 326 ts_tablet_manager.cc:1367] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9: Time spent bootstrapping tablet: real 0.302s user 0.049s sys 0.009s
I20240503 08:13:13.507427 433 tablet_copy_client.cc:308] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:13:13.508466 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 4babf0ea9b954a23b1b1c4e403dce0bd from peer 4574396f35514c9caccb346f1a7af84d ({username='slave'} at 127.20.107.131:42119)
I20240503 08:13:13.508646 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 4babf0ea9b954a23b1b1c4e403dce0bd from peer 4574396f35514c9caccb346f1a7af84d at {username='slave'} at 127.20.107.131:42119: session id = 4574396f35514c9caccb346f1a7af84d-4babf0ea9b954a23b1b1c4e403dce0bd
I20240503 08:13:13.510941 30675 raft_consensus.cc:1203] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:13.511080 369 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap complete.
I20240503 08:13:13.511619 28785 tablet_copy_source_session.cc:205] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.511675 369 ts_tablet_manager.cc:1367] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: Time spent bootstrapping tablet: real 0.263s user 0.055s sys 0.003s
I20240503 08:13:13.512748 369 raft_consensus.cc:348] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:13.513751 32745 tablet_copy_client.cc:308] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:13:13.513900 369 raft_consensus.cc:729] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Initialized, Role: LEARNER
I20240503 08:13:13.514178 369 consensus_queue.cc:260] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:13.514449 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet b4ceeeb3495e4a8fac6e13e431c64af4 from peer b29361eb4afa4ab3af7d8019d8a5042d ({username='slave'} at 127.20.107.136:48299)
I20240503 08:13:13.514613 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet b4ceeeb3495e4a8fac6e13e431c64af4 from peer b29361eb4afa4ab3af7d8019d8a5042d at {username='slave'} at 127.20.107.136:48299: session id = b29361eb4afa4ab3af7d8019d8a5042d-b4ceeeb3495e4a8fac6e13e431c64af4
I20240503 08:13:13.515166 369 ts_tablet_manager.cc:1398] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:13:13.516096 433 ts_tablet_manager.cc:1875] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:13:13.516541 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session bea2145b8e24436392b18ec3cfe90b93-0caf86850b3841ea811b0a6ab19ae330 received from {username='slave'} at 127.20.107.135:43079
I20240503 08:13:13.516676 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-0caf86850b3841ea811b0a6ab19ae330 on tablet 0caf86850b3841ea811b0a6ab19ae330 with peer bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:13.517583 32275 raft_consensus.cc:1050] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER bea2145b8e24436392b18ec3cfe90b93 to VOTER
I20240503 08:13:13.518615 326 raft_consensus.cc:348] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:13.518817 326 raft_consensus.cc:729] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8375219addc48fa99ac474ed1c9aef9, State: Initialized, Role: LEARNER
I20240503 08:13:13.518662 32275 consensus_queue.cc:237] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:13:13.519070 326 consensus_queue.cc:260] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:13.521373 28655 tablet_copy_source_session.cc:205] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.524046 30543 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:13.524420 28630 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:13.524999 32661 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:13.525944 28502 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:13.527823 32745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ceeeb3495e4a8fac6e13e431c64af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.529129 32661 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:13.531091 32559 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:13.531332 28505 consensus_queue.cc:237] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:13.559824 385 tablet_copy_client.cc:308] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:13.560784 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet e9997e8da99c4f45a63bc3db521b33ba from peer 21d21a5b596b4d48a17a1fb54a972844 ({username='slave'} at 127.20.107.134:32851)
I20240503 08:13:13.560981 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet e9997e8da99c4f45a63bc3db521b33ba from peer 21d21a5b596b4d48a17a1fb54a972844 at {username='slave'} at 127.20.107.134:32851: session id = 21d21a5b596b4d48a17a1fb54a972844-e9997e8da99c4f45a63bc3db521b33ba
I20240503 08:13:13.564047 28394 tablet_copy_source_session.cc:205] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.566144 385 ts_tablet_manager.cc:1875] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:13:13.570106 417 tablet_copy_client.cc:308] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:13:13.571013 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 4babf0ea9b954a23b1b1c4e403dce0bd from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:43079)
I20240503 08:13:13.571249 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 4babf0ea9b954a23b1b1c4e403dce0bd from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:43079: session id = bea2145b8e24436392b18ec3cfe90b93-4babf0ea9b954a23b1b1c4e403dce0bd
I20240503 08:13:13.575547 28785 tablet_copy_source_session.cc:205] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.581084 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4babf0ea9b954a23b1b1c4e403dce0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.582440 326 ts_tablet_manager.cc:1398] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9: Time spent starting tablet: real 0.109s user 0.010s sys 0.027s
I20240503 08:13:13.583400 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session c8375219addc48fa99ac474ed1c9aef9-564fb1b51abd4f8da0372ec99e3c599c received from {username='slave'} at 127.20.107.133:41339
I20240503 08:13:13.583662 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session c8375219addc48fa99ac474ed1c9aef9-564fb1b51abd4f8da0372ec99e3c599c on tablet 564fb1b51abd4f8da0372ec99e3c599c with peer c8375219addc48fa99ac474ed1c9aef9
I20240503 08:13:13.588152 433 ts_tablet_manager.cc:1888] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:13:13.588503 433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4babf0ea9b954a23b1b1c4e403dce0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.594286 417 tablet_copy_client.cc:783] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.594712 417 tablet_copy_client.cc:647] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.598181 417 tablet_copy_client.cc:523] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.605284 417 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
I20240503 08:13:13.608523 30675 raft_consensus.cc:1261] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:13.609382 420 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:13.618566 28761 raft_consensus.cc:1261] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:13.619207 32682 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:13:13.621194 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac392ff7965a405f96efc777aca4f0bb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.625900 385 ts_tablet_manager.cc:1888] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:13:13.625677 32682 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:13.626235 385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9997e8da99c4f45a63bc3db521b33ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.634104 30675 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:13.636405 32559 raft_consensus.cc:1050] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 to VOTER
I20240503 08:13:13.637216 433 tablet_copy_client.cc:783] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.637385 32559 consensus_queue.cc:237] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:13.637591 433 tablet_copy_client.cc:647] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.641053 30412 raft_consensus.cc:1203] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:13.641880 385 tablet_copy_client.cc:783] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.642427 385 tablet_copy_client.cc:647] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.643316 433 tablet_copy_client.cc:523] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.647042 28761 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:13:13.650228 433 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:13:13.650193 30412 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:13.650919 32275 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:13.655333 30671 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:13.656342 32661 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:13.658161 28762 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:13.658768 32661 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:13.665161 369 tablet_replica.cc:330] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: stopping tablet replica
I20240503 08:13:13.665594 369 raft_consensus.cc:2229] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 08:13:13.665809 369 raft_consensus.cc:2258] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 08:13:13.667135 369 ts_tablet_manager.cc:1875] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:13.675678 28227 catalog_manager.cc:5484] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 6 to 7, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:13.695039 369 ts_tablet_manager.cc:1888] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:13:13.695374 369 log.cc:1196] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal/wals/0caf86850b3841ea811b0a6ab19ae330
I20240503 08:13:13.696076 369 ts_tablet_manager.cc:902] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:13:13.713621 385 tablet_copy_client.cc:523] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.717537 385 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap starting.
I20240503 08:13:13.728911 470 ts_tablet_manager.cc:902] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:13:13.732740 32661 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:13.744366 28762 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:13.751852 32275 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:13.756374 28505 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:13.757102 30543 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:13.760187 30412 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:13.761492 30675 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:13.771967 28228 catalog_manager.cc:5484] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93 reported cstate change: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:13.787951 28630 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:13:13.795147 28224 catalog_manager.cc:5484] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:13.838485 470 tablet_copy_client.cc:308] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:13:13.839462 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 0caf86850b3841ea811b0a6ab19ae330 from peer 4574396f35514c9caccb346f1a7af84d ({username='slave'} at 127.20.107.131:42119)
I20240503 08:13:13.839910 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 0caf86850b3841ea811b0a6ab19ae330 from peer 4574396f35514c9caccb346f1a7af84d at {username='slave'} at 127.20.107.131:42119: session id = 4574396f35514c9caccb346f1a7af84d-0caf86850b3841ea811b0a6ab19ae330
I20240503 08:13:13.843089 28785 tablet_copy_source_session.cc:205] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.852557 470 ts_tablet_manager.cc:1875] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:13:13.862686 369 tablet_copy_client.cc:308] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:13:13.863683 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 0caf86850b3841ea811b0a6ab19ae330 from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:43079)
I20240503 08:13:13.863965 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 0caf86850b3841ea811b0a6ab19ae330 from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:43079: session id = bea2145b8e24436392b18ec3cfe90b93-0caf86850b3841ea811b0a6ab19ae330
I20240503 08:13:13.864980 32745 tablet_copy_client.cc:783] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.865288 32745 tablet_copy_client.cc:647] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.865631 30278 raft_consensus.cc:1203] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:13.867233 32745 tablet_copy_client.cc:523] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.876724 326 ts_tablet_manager.cc:902] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:13:13.881671 28785 tablet_copy_source_session.cc:205] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.884709 369 ts_tablet_manager.cc:1875] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:13:13.887961 470 ts_tablet_manager.cc:1888] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:13:13.892064 32722 tablet_bootstrap.cc:492] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:13.901221 32722 tablet_bootstrap.cc:492] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap complete.
I20240503 08:13:13.901881 32722 ts_tablet_manager.cc:1367] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844: Time spent bootstrapping tablet: real 0.586s user 0.064s sys 0.024s
I20240503 08:13:13.908943 32745 tablet_bootstrap.cc:492] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap starting.
I20240503 08:13:13.913730 470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caf86850b3841ea811b0a6ab19ae330. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.902822 32722 raft_consensus.cc:348] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:13.917281 32722 raft_consensus.cc:729] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Initialized, Role: LEARNER
I20240503 08:13:13.918292 32722 consensus_queue.cc:260] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:13.918968 32722 ts_tablet_manager.cc:1398] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844: Time spent starting tablet: real 0.017s user 0.000s sys 0.001s
I20240503 08:13:13.921058 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session 21d21a5b596b4d48a17a1fb54a972844-173376257c4946b5a801a27e4dffe90d received from {username='slave'} at 127.20.107.134:32851
I20240503 08:13:13.921204 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session 21d21a5b596b4d48a17a1fb54a972844-173376257c4946b5a801a27e4dffe90d on tablet 173376257c4946b5a801a27e4dffe90d with peer 21d21a5b596b4d48a17a1fb54a972844
I20240503 08:13:13.930063 470 tablet_copy_client.cc:783] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.930439 470 tablet_copy_client.cc:647] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.935809 470 tablet_copy_client.cc:523] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.941233 508 ts_tablet_manager.cc:902] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:13:13.942028 470 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:13:13.942207 369 ts_tablet_manager.cc:1888] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240503 08:13:13.943042 369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caf86850b3841ea811b0a6ab19ae330. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.947962 508 tablet_copy_client.cc:308] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:13:13.948796 28524 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet a20489bf65774902af38d228ee966ffe from peer b29361eb4afa4ab3af7d8019d8a5042d ({username='slave'} at 127.20.107.136:33075)
I20240503 08:13:13.949034 28524 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet a20489bf65774902af38d228ee966ffe from peer b29361eb4afa4ab3af7d8019d8a5042d at {username='slave'} at 127.20.107.136:33075: session id = b29361eb4afa4ab3af7d8019d8a5042d-a20489bf65774902af38d228ee966ffe
I20240503 08:13:13.953375 28524 tablet_copy_source_session.cc:205] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.970698 326 tablet_copy_client.cc:308] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:13:13.973688 28524 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet a20489bf65774902af38d228ee966ffe from peer c8375219addc48fa99ac474ed1c9aef9 ({username='slave'} at 127.20.107.133:43991)
I20240503 08:13:13.973986 28524 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet a20489bf65774902af38d228ee966ffe from peer c8375219addc48fa99ac474ed1c9aef9 at {username='slave'} at 127.20.107.133:43991: session id = c8375219addc48fa99ac474ed1c9aef9-a20489bf65774902af38d228ee966ffe
I20240503 08:13:13.977548 28524 tablet_copy_source_session.cc:205] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.978283 369 tablet_copy_client.cc:783] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.978608 369 tablet_copy_client.cc:647] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.978955 326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20489bf65774902af38d228ee966ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.981350 369 tablet_copy_client.cc:523] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.987625 326 tablet_copy_client.cc:783] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.987910 326 tablet_copy_client.cc:647] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.990922 326 tablet_copy_client.cc:523] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.997404 369 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
I20240503 08:13:14.005813 326 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9: Bootstrap starting.
I20240503 08:13:14.008684 417 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:14.017154 417 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: Bootstrap complete.
I20240503 08:13:14.018229 417 ts_tablet_manager.cc:1367] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: Time spent bootstrapping tablet: real 0.413s user 0.060s sys 0.006s
I20240503 08:13:14.019734 533 ts_tablet_manager.cc:902] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:13:14.020985 417 raft_consensus.cc:348] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:14.021718 417 raft_consensus.cc:729] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Initialized, Role: LEARNER
I20240503 08:13:14.022014 417 consensus_queue.cc:260] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:14.023896 417 ts_tablet_manager.cc:1398] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 08:13:14.024617 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session bea2145b8e24436392b18ec3cfe90b93-4babf0ea9b954a23b1b1c4e403dce0bd received from {username='slave'} at 127.20.107.135:43079
I20240503 08:13:14.024780 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-4babf0ea9b954a23b1b1c4e403dce0bd on tablet 4babf0ea9b954a23b1b1c4e403dce0bd with peer bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:14.056181 533 tablet_copy_client.cc:308] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:13:14.068682 508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20489bf65774902af38d228ee966ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:14.073055 28524 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet ac392ff7965a405f96efc777aca4f0bb from peer c8375219addc48fa99ac474ed1c9aef9 ({username='slave'} at 127.20.107.133:43991)
I20240503 08:13:14.074034 28524 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet ac392ff7965a405f96efc777aca4f0bb from peer c8375219addc48fa99ac474ed1c9aef9 at {username='slave'} at 127.20.107.133:43991: session id = c8375219addc48fa99ac474ed1c9aef9-ac392ff7965a405f96efc777aca4f0bb
I20240503 08:13:14.074627 30412 raft_consensus.cc:1203] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:14.079198 28524 tablet_copy_source_session.cc:205] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:14.080816 533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac392ff7965a405f96efc777aca4f0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:14.090302 533 tablet_copy_client.cc:783] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:14.090682 533 tablet_copy_client.cc:647] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:14.093192 533 tablet_copy_client.cc:523] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:14.101441 32682 raft_consensus.cc:1050] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d to VOTER
I20240503 08:13:14.102607 32682 consensus_queue.cc:237] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:14.121320 28762 raft_consensus.cc:1261] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:14.122419 32684 consensus_queue.cc:1035] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:14.122773 28630 raft_consensus.cc:1261] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:14.124486 30672 raft_consensus.cc:1261] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:14.126191 32684 consensus_queue.cc:1035] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:14.126905 32684 consensus_queue.cc:1035] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:14.170645 32682 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:14.194774 28631 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:14.204031 28232 catalog_manager.cc:5484] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:14.214283 433 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:14.214872 433 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: Bootstrap complete.
I20240503 08:13:14.215217 433 ts_tablet_manager.cc:1367] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.565s user 0.068s sys 0.001s
I20240503 08:13:14.215785 28762 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:14.216123 433 raft_consensus.cc:348] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:14.216387 433 raft_consensus.cc:729] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: LEARNER
I20240503 08:13:14.218932 32328 raft_consensus.cc:1050] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 to VOTER
I20240503 08:13:14.219658 32328 consensus_queue.cc:237] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:14.216725 433 consensus_queue.cc:260] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:14.245296 533 tablet_bootstrap.cc:492] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9: Bootstrap starting.
I20240503 08:13:14.246451 433 ts_tablet_manager.cc:1398] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.031s user 0.004s sys 0.000s
I20240503 08:13:14.249310 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session 4574396f35514c9caccb346f1a7af84d-4babf0ea9b954a23b1b1c4e403dce0bd received from {username='slave'} at 127.20.107.131:42119
I20240503 08:13:14.249450 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session 4574396f35514c9caccb346f1a7af84d-4babf0ea9b954a23b1b1c4e403dce0bd on tablet 4babf0ea9b954a23b1b1c4e403dce0bd with peer 4574396f35514c9caccb346f1a7af84d
I20240503 08:13:14.254197 28630 raft_consensus.cc:1203] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:13:14.265090 30412 raft_consensus.cc:1261] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:14.265938 343 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:14.279256 30278 raft_consensus.cc:1261] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:14.279911 343 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:14.287865 32716 raft_consensus.cc:1050] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER c8375219addc48fa99ac474ed1c9aef9 to VOTER
I20240503 08:13:14.290071 32716 consensus_queue.cc:237] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:14.291437 28374 raft_consensus.cc:1261] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:14.292196 343 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:14.319872 30672 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:14.329375 343 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:14.330340 28505 consensus_queue.cc:237] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:14.337230 28630 raft_consensus.cc:1261] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:14.340744 28374 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:14.344470 326 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:14.344996 326 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9: Bootstrap complete.
I20240503 08:13:14.345315 326 ts_tablet_manager.cc:1367] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9: Time spent bootstrapping tablet: real 0.340s user 0.050s sys 0.020s
I20240503 08:13:14.346180 326 raft_consensus.cc:348] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:14.346487 326 raft_consensus.cc:729] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8375219addc48fa99ac474ed1c9aef9, State: Initialized, Role: LEARNER
I20240503 08:13:14.346727 326 consensus_queue.cc:260] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:14.346788 420 consensus_queue.cc:1035] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:14.349541 28228 catalog_manager.cc:5484] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:14.351975 326 ts_tablet_manager.cc:1398] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 08:13:14.353824 30278 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:14.358891 28524 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session c8375219addc48fa99ac474ed1c9aef9-a20489bf65774902af38d228ee966ffe received from {username='slave'} at 127.20.107.133:43991
I20240503 08:13:14.359097 28524 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session c8375219addc48fa99ac474ed1c9aef9-a20489bf65774902af38d228ee966ffe on tablet a20489bf65774902af38d228ee966ffe with peer c8375219addc48fa99ac474ed1c9aef9
I20240503 08:13:14.359290 30281 raft_consensus.cc:1261] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:14.360196 32718 consensus_queue.cc:1035] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:14.362320 542 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:14.364136 28373 raft_consensus.cc:1261] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:14.365440 32709 consensus_queue.cc:1035] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:14.370383 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet da529c8f8de241d38678d1596b9402ac with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:14.373904 28630 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:14.374205 28232 catalog_manager.cc:5484] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:14.393375 32718 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:14.396365 28373 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:14.404510 28227 catalog_manager.cc:5484] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:14.417794 28761 consensus_queue.cc:237] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:14.421519 30281 raft_consensus.cc:1261] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:14.422528 32709 consensus_queue.cc:1035] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:14.424458 30281 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:14.427711 30543 raft_consensus.cc:1203] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:13:14.447825 28744 tablet_service.cc:1514] Processing DeleteTablet for tablet da529c8f8de241d38678d1596b9402ac with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 6)) from {username='slave'} at 127.0.0.1:39918
I20240503 08:13:14.395241 30412 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:14.457263 32716 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:14.464125 28631 raft_consensus.cc:1261] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:14.478513 28373 raft_consensus.cc:1261] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:14.479554 32718 consensus_queue.cc:1035] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:14.481564 30675 raft_consensus.cc:1261] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:14.485352 575 tablet_replica.cc:330] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb: stopping tablet replica
I20240503 08:13:14.485203 32682 consensus_queue.cc:1035] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:14.485852 575 raft_consensus.cc:2229] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:14.486097 575 raft_consensus.cc:2258] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:14.488772 30672 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:14.499358 575 ts_tablet_manager.cc:1875] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:14.504304 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 564fb1b51abd4f8da0372ec99e3c599c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:14.507231 30278 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:14.517227 28224 catalog_manager.cc:5484] T 564fb1b51abd4f8da0372ec99e3c599c P c8375219addc48fa99ac474ed1c9aef9 reported cstate change: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:14.516839 508 tablet_copy_client.cc:783] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:14.518239 508 tablet_copy_client.cc:647] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:14.521714 28631 consensus_queue.cc:237] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:14.523559 28614 tablet_service.cc:1514] Processing DeleteTablet for tablet 564fb1b51abd4f8da0372ec99e3c599c with delete_type TABLET_DATA_TOMBSTONED (TS 4574396f35514c9caccb346f1a7af84d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56340
I20240503 08:13:14.526412 30414 raft_consensus.cc:1261] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:14.527705 30278 raft_consensus.cc:1261] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:14.528200 32328 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:14.528759 508 tablet_copy_client.cc:523] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:14.528903 562 consensus_queue.cc:1035] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:14.532450 562 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:14.536249 30278 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:14.546706 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 585ba864cc384e7f9db90e9d4c47e82c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:14.547906 28224 catalog_manager.cc:5484] T 585ba864cc384e7f9db90e9d4c47e82c P c8375219addc48fa99ac474ed1c9aef9 reported cstate change: config changed from index 8 to 9, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:14.547900 30414 raft_consensus.cc:2932] T 585ba864cc384e7f9db90e9d4c47e82c P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:14.563194 369 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:14.563937 369 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap complete.
I20240503 08:13:14.564730 369 ts_tablet_manager.cc:1367] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: Time spent bootstrapping tablet: real 0.567s user 0.125s sys 0.121s
I20240503 08:13:14.565924 369 raft_consensus.cc:348] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:13:14.566340 369 raft_consensus.cc:729] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Initialized, Role: LEARNER
I20240503 08:13:14.566645 369 consensus_queue.cc:260] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:13:14.567373 369 ts_tablet_manager.cc:1398] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:13:14.568406 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session bea2145b8e24436392b18ec3cfe90b93-0caf86850b3841ea811b0a6ab19ae330 received from {username='slave'} at 127.20.107.135:43079
I20240503 08:13:14.568565 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-0caf86850b3841ea811b0a6ab19ae330 on tablet 0caf86850b3841ea811b0a6ab19ae330 with peer bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:14.574394 28353 tablet_service.cc:1514] Processing DeleteTablet for tablet 585ba864cc384e7f9db90e9d4c47e82c with delete_type TABLET_DATA_TOMBSTONED (TS 4e4a1242f3714d03a6c9ee51dd63929c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51022
I20240503 08:13:14.575902 28373 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:14.585040 470 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:14.593901 508 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap starting.
I20240503 08:13:14.602874 470 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: Bootstrap complete.
I20240503 08:13:14.610263 470 ts_tablet_manager.cc:1367] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.668s user 0.057s sys 0.004s
I20240503 08:13:14.592249 583 tablet_replica.cc:330] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d: stopping tablet replica
I20240503 08:13:14.611061 583 raft_consensus.cc:2229] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:14.611629 583 raft_consensus.cc:2258] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:14.612437 583 ts_tablet_manager.cc:1875] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:14.610810 385 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:14.613605 385 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap complete.
I20240503 08:13:14.613950 385 ts_tablet_manager.cc:1367] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: Time spent bootstrapping tablet: real 0.897s user 0.057s sys 0.011s
I20240503 08:13:14.615116 385 raft_consensus.cc:348] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:14.615460 385 raft_consensus.cc:729] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Initialized, Role: LEARNER
I20240503 08:13:14.615774 385 consensus_queue.cc:260] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:14.616518 385 ts_tablet_manager.cc:1398] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:14.617432 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session 21d21a5b596b4d48a17a1fb54a972844-e9997e8da99c4f45a63bc3db521b33ba received from {username='slave'} at 127.20.107.134:32851
I20240503 08:13:14.617604 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session 21d21a5b596b4d48a17a1fb54a972844-e9997e8da99c4f45a63bc3db521b33ba on tablet e9997e8da99c4f45a63bc3db521b33ba with peer 21d21a5b596b4d48a17a1fb54a972844
I20240503 08:13:14.620780 32275 raft_consensus.cc:1050] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 to VOTER
I20240503 08:13:14.621639 592 tablet_replica.cc:330] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c: stopping tablet replica
I20240503 08:13:14.621948 592 raft_consensus.cc:2229] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:14.621814 32275 consensus_queue.cc:237] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:14.624959 592 raft_consensus.cc:2258] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:14.629635 30412 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:14.629994 28505 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:14.632512 470 raft_consensus.cc:348] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:13:14.630913 491 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:14.638325 32559 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:14.661944 592 ts_tablet_manager.cc:1875] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:14.675125 470 raft_consensus.cc:729] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: LEARNER
I20240503 08:13:14.675619 470 consensus_queue.cc:260] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:13:14.677006 30543 raft_consensus.cc:1203] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:13:14.677536 470 ts_tablet_manager.cc:1398] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.067s user 0.002s sys 0.000s
I20240503 08:13:14.681473 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session 4574396f35514c9caccb346f1a7af84d-0caf86850b3841ea811b0a6ab19ae330 received from {username='slave'} at 127.20.107.131:42119
I20240503 08:13:14.681511 28630 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:14.681623 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session 4574396f35514c9caccb346f1a7af84d-0caf86850b3841ea811b0a6ab19ae330 on tablet 0caf86850b3841ea811b0a6ab19ae330 with peer 4574396f35514c9caccb346f1a7af84d
I20240503 08:13:14.687664 32559 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:14.695252 28232 catalog_manager.cc:5484] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:14.702194 30412 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:14.708282 32559 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:14.710638 28630 raft_consensus.cc:1203] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:13:14.733984 575 ts_tablet_manager.cc:1888] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:14.734387 28631 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:14.774091 30278 raft_consensus.cc:1203] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:13:14.783273 28505 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:14.795416 30412 raft_consensus.cc:1203] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 08:13:14.765805 32716 raft_consensus.cc:1050] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER 4574396f35514c9caccb346f1a7af84d to VOTER
I20240503 08:13:14.804669 32716 consensus_queue.cc:237] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:14.813326 575 log.cc:1196] T da529c8f8de241d38678d1596b9402ac P 93f7e733cec9415eb422b0a7ba8621cb: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/wals/da529c8f8de241d38678d1596b9402ac
I20240503 08:13:14.815006 28218 catalog_manager.cc:4830] TS 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095): tablet da529c8f8de241d38678d1596b9402ac (table rebalance_test_table_9 [id=e4d3a510e86a4cbe8d3d611eac908187]) successfully deleted
I20240503 08:13:14.834666 30412 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:14.834820 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:14.835355 32718 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:14.855149 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:14.860410 583 ts_tablet_manager.cc:1888] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:13:14.860603 583 log.cc:1196] T 564fb1b51abd4f8da0372ec99e3c599c P 4574396f35514c9caccb346f1a7af84d: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/564fb1b51abd4f8da0372ec99e3c599c
I20240503 08:13:14.872107 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:14.889381 30523 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:14.890777 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:14.915679 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:14.916787 28505 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:14.917519 32716 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:14.935189 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:14.944325 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:14.868438 28615 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:14.868568 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:14.858031 28631 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:14.947732 30542 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:14.947968 32716 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:14.952379 32716 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:14.954007 28217 catalog_manager.cc:4830] TS 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): tablet 564fb1b51abd4f8da0372ec99e3c599c (table rebalance_test_table_7 [id=0225804a5c4a42cbb32c4b0029752bd1]) successfully deleted
I20240503 08:13:14.961948 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:14.965663 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:14.987996 32718 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:14.993322 28505 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:14.999944 592 ts_tablet_manager.cc:1888] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:15.000206 592 log.cc:1196] T 585ba864cc384e7f9db90e9d4c47e82c P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/585ba864cc384e7f9db90e9d4c47e82c
I20240503 08:13:15.001071 30412 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:15.012027 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:15.035660 32745 tablet_bootstrap.cc:492] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:15.036084 28218 catalog_manager.cc:4830] TS 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): tablet 585ba864cc384e7f9db90e9d4c47e82c (table rebalance_test_table_6 [id=87e8ca2152ac403cb71e5eb67c0b2e0e]) successfully deleted
I20240503 08:13:15.039526 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:15.040752 32745 tablet_bootstrap.cc:492] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap complete.
I20240503 08:13:15.041128 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:15.043036 32745 ts_tablet_manager.cc:1367] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent bootstrapping tablet: real 1.134s user 0.066s sys 0.003s
I20240503 08:13:15.043980 32745 raft_consensus.cc:348] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:15.044608 28631 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:15.048493 32745 raft_consensus.cc:729] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b29361eb4afa4ab3af7d8019d8a5042d, State: Initialized, Role: LEARNER
I20240503 08:13:15.057910 28231 catalog_manager.cc:5484] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 8 to 9, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:15.065217 28762 consensus_queue.cc:237] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } }
I20240503 08:13:15.065739 32745 consensus_queue.cc:260] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:15.066603 32745 ts_tablet_manager.cc:1398] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240503 08:13:15.069792 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-b4ceeeb3495e4a8fac6e13e431c64af4 received from {username='slave'} at 127.20.107.136:48299
I20240503 08:13:15.070011 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-b4ceeeb3495e4a8fac6e13e431c64af4 on tablet b4ceeeb3495e4a8fac6e13e431c64af4 with peer b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:15.072683 28630 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:15.074288 642 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:15.079267 30414 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:15.081360 642 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:15.089888 30523 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:15.113653 32718 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.119778 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4babf0ea9b954a23b1b1c4e403dce0bd with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:15.122468 28630 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.125584 28232 catalog_manager.cc:5484] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 9 to 10, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:15.134253 30412 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.144811 28762 consensus_queue.cc:237] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } }
I20240503 08:13:15.148245 28630 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:15.149019 32718 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:15.152890 28505 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:15.156674 28631 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.179986 30523 tablet_service.cc:1514] Processing DeleteTablet for tablet 4babf0ea9b954a23b1b1c4e403dce0bd with delete_type TABLET_DATA_TOMBSTONED (TS bea2145b8e24436392b18ec3cfe90b93 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56760
I20240503 08:13:15.191514 491 raft_consensus.cc:1050] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 to VOTER
I20240503 08:13:15.192538 491 consensus_queue.cc:237] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:15.200124 30414 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:15.200675 28225 catalog_manager.cc:5484] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 10 to 11, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.206560 28484 tablet_service.cc:1514] Processing DeleteTablet for tablet 4babf0ea9b954a23b1b1c4e403dce0bd with delete_type TABLET_DATA_TOMBSTONED (TS 716ffbb2a00b4f43b03e5cedcc9d302a not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:54162
I20240503 08:13:15.226554 420 raft_consensus.cc:1050] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER c8375219addc48fa99ac474ed1c9aef9 to VOTER
I20240503 08:13:15.227831 420 consensus_queue.cc:237] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:15.231353 30672 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:15.231966 30412 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:15.232017 32275 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:15.233464 32559 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:15.235420 666 tablet_replica.cc:330] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: stopping tablet replica
I20240503 08:13:15.235574 28505 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:15.235823 666 raft_consensus.cc:2229] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:15.236410 666 raft_consensus.cc:2258] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:15.236905 32559 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:15.238014 666 ts_tablet_manager.cc:1875] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:15.266180 533 tablet_bootstrap.cc:492] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:15.154561 643 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.288113 663 tablet_replica.cc:330] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: stopping tablet replica
I20240503 08:13:15.288419 663 raft_consensus.cc:2229] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 08:13:15.288689 663 pending_rounds.cc:62] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: Trying to abort 1 pending ops.
I20240503 08:13:15.288851 663 pending_rounds.cc:68] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: Aborting op as it isn't in flight: id { term: 1 index: 9 } timestamp: 7023509482708910080 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "4babf0ea9b954a23b1b1c4e403dce0bd" old_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } } new_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } } }
I20240503 08:13:15.289706 663 raft_consensus.cc:2894] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Aborting config change with OpId 1.9: Aborted: Op aborted
I20240503 08:13:15.289942 663 raft_consensus.cc:2866] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:13:15.290127 663 raft_consensus.cc:2258] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 08:13:15.307123 533 tablet_bootstrap.cc:492] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9: Bootstrap complete.
I20240503 08:13:15.307790 533 ts_tablet_manager.cc:1367] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9: Time spent bootstrapping tablet: real 1.063s user 0.067s sys 0.009s
I20240503 08:13:15.309322 642 raft_consensus.cc:1050] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER bea2145b8e24436392b18ec3cfe90b93 to VOTER
I20240503 08:13:15.309854 32718 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.107s
I20240503 08:13:15.320399 30414 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
W20240503 08:13:15.330557 28308 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.134:34587, user_credentials={real_user=slave}} blocked reactor thread for 62266.5us
I20240503 08:13:15.333071 30672 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:15.332095 673 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:15.343025 28228 catalog_manager.cc:5484] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d reported cstate change: config changed from index 9 to 10, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:15.349197 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4babf0ea9b954a23b1b1c4e403dce0bd with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 08:13:15.362345 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:15.367533 642 consensus_queue.cc:237] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
W20240503 08:13:15.368378 28439 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.136:42389, user_credentials={real_user=slave}} blocked reactor thread for 71158.4us
W20240503 08:13:15.376416 672 log.cc:924] Time spent T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20240503 08:13:15.378968 28505 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:15.363246 30278 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:15.388137 32682 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240503 08:13:15.405655 30412 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:15.406596 676 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:15.415545 30543 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:15.416353 676 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:15.419157 30672 raft_consensus.cc:1203] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:15.428992 28631 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:15.431972 32716 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:15.432212 666 ts_tablet_manager.cc:1888] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:13:15.432428 666 log.cc:1196] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/4babf0ea9b954a23b1b1c4e403dce0bd
I20240503 08:13:15.433602 28218 catalog_manager.cc:4830] TS 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): tablet 4babf0ea9b954a23b1b1c4e403dce0bd (table rebalance_test_table_5 [id=311fe8a0590b44a8a07f0203d8f65e32]) successfully deleted
I20240503 08:13:15.435415 28505 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:15.436357 32709 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:15.449968 30414 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:15.453078 533 raft_consensus.cc:348] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:15.454016 533 raft_consensus.cc:729] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8375219addc48fa99ac474ed1c9aef9, State: Initialized, Role: LEARNER
I20240503 08:13:15.456753 32709 raft_consensus.cc:1013] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: attempt to promote peer 4574396f35514c9caccb346f1a7af84d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:13:15.463778 676 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:15.473264 30412 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:15.494272 28227 catalog_manager.cc:5484] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: config changed from index 8 to 9, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:15.504654 28631 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:15.481925 533 consensus_queue.cc:260] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:13:15.521770 32716 raft_consensus.cc:1050] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER 4574396f35514c9caccb346f1a7af84d to VOTER
I20240503 08:13:15.523295 32716 consensus_queue.cc:237] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } }
I20240503 08:13:15.528138 28631 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 08:13:15.528299 28569 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.129:38529, user_credentials={real_user=slave}} blocked reactor thread for 110711us
I20240503 08:13:15.530537 30412 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:15.531251 32709 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:15.542340 28374 consensus_queue.cc:237] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:15.554898 663 ts_tablet_manager.cc:1875] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:15.557301 30543 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:15.558094 676 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:15.559022 676 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:15.561806 32709 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.561807 30543 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:15.567071 30413 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.573379 30542 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:15.576292 28226 catalog_manager.cc:5484] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: config changed from index 9 to 10, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.578110 28505 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:15.574069 28631 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.593457 673 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:15.596819 28631 consensus_queue.cc:237] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:15.603199 28374 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:15.604285 30412 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:15.606084 28501 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:15.606549 32682 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.013s
I20240503 08:13:15.608659 491 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:15.609779 533 ts_tablet_manager.cc:1398] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9: Time spent starting tablet: real 0.209s user 0.002s sys 0.000s
I20240503 08:13:15.610983 32275 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:15.614778 28524 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session c8375219addc48fa99ac474ed1c9aef9-ac392ff7965a405f96efc777aca4f0bb received from {username='slave'} at 127.20.107.133:43991
I20240503 08:13:15.614975 28524 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session c8375219addc48fa99ac474ed1c9aef9-ac392ff7965a405f96efc777aca4f0bb on tablet ac392ff7965a405f96efc777aca4f0bb with peer c8375219addc48fa99ac474ed1c9aef9
I20240503 08:13:15.615762 542 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
I20240503 08:13:15.620922 28231 catalog_manager.cc:5484] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 10 to 11, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:15.623602 28502 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:15.624290 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e9997e8da99c4f45a63bc3db521b33ba with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:15.626055 32716 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:15.626696 28501 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:15.627808 28502 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.633661 30278 raft_consensus.cc:1203] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:15.637452 30656 tablet_service.cc:1514] Processing DeleteTablet for tablet e9997e8da99c4f45a63bc3db521b33ba with delete_type TABLET_DATA_TOMBSTONED (TS b29361eb4afa4ab3af7d8019d8a5042d not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:53250
I20240503 08:13:15.642762 30412 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:15.659567 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:15.672559 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:15.690940 30278 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:15.691757 678 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:15.708019 508 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:15.709241 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:15.708699 508 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap complete.
I20240503 08:13:15.710022 508 ts_tablet_manager.cc:1367] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d: Time spent bootstrapping tablet: real 1.116s user 0.058s sys 0.014s
I20240503 08:13:15.723814 32682 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:15.730103 30542 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:15.732645 28231 catalog_manager.cc:5484] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 8 to 9, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:15.735574 28373 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:15.736603 683 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:15.781296 663 ts_tablet_manager.cc:1888] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:13:15.781929 663 log.cc:1196] T 4babf0ea9b954a23b1b1c4e403dce0bd P bea2145b8e24436392b18ec3cfe90b93: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-6/wal/wals/4babf0ea9b954a23b1b1c4e403dce0bd
I20240503 08:13:15.822939 508 raft_consensus.cc:348] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:15.823377 508 raft_consensus.cc:729] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b29361eb4afa4ab3af7d8019d8a5042d, State: Initialized, Role: LEARNER
I20240503 08:13:15.823628 508 consensus_queue.cc:260] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:15.824355 508 ts_tablet_manager.cc:1398] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d: Time spent starting tablet: real 0.114s user 0.001s sys 0.000s
I20240503 08:13:15.824894 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:15.837510 30278 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:15.852823 28501 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:15.857199 28353 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:15.869246 28219 catalog_manager.cc:4830] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): tablet 4babf0ea9b954a23b1b1c4e403dce0bd (table rebalance_test_table_5 [id=311fe8a0590b44a8a07f0203d8f65e32]) successfully deleted
I20240503 08:13:15.881840 678 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:15.885408 683 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:15.881847 28373 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:15.890242 28228 catalog_manager.cc:5484] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:15.898653 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:15.900849 30523 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:15.901505 28502 consensus_queue.cc:237] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:15.905469 30281 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:15.905344 30278 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:15.906551 30543 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:15.906347 28374 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:15.913328 32682 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:15.930150 28762 consensus_queue.cc:237] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } }
I20240503 08:13:15.933115 30543 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.933904 28631 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:15.933991 675 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:15.934664 642 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:15.937759 30412 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:15.938118 30542 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:15.938513 642 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:15.941263 642 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.945384 32682 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:15.945293 30414 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.948695 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a20489bf65774902af38d228ee966ffe with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:15.948184 30281 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:15.950423 28226 catalog_manager.cc:5484] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 9 to 10, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:15.955767 28232 catalog_manager.cc:5484] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: config changed from index 10 to 11, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.961722 28353 tablet_service.cc:1514] Processing DeleteTablet for tablet a20489bf65774902af38d228ee966ffe with delete_type TABLET_DATA_TOMBSTONED (TS 4e4a1242f3714d03a6c9ee51dd63929c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51022
I20240503 08:13:15.978546 28484 tablet_service.cc:1514] Processing DeleteTablet for tablet 0caf86850b3841ea811b0a6ab19ae330 with delete_type TABLET_DATA_TOMBSTONED (TS 716ffbb2a00b4f43b03e5cedcc9d302a not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:54162
I20240503 08:13:15.982916 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:15.991647 30542 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:15.995131 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:15.994081 28631 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:15.998840 712 tablet_replica.cc:330] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d: stopping tablet replica
I20240503 08:13:15.999203 28524 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-a20489bf65774902af38d228ee966ffe received from {username='slave'} at 127.20.107.136:33075
I20240503 08:13:15.999374 28524 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-a20489bf65774902af38d228ee966ffe on tablet a20489bf65774902af38d228ee966ffe with peer b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:15.999992 712 raft_consensus.cc:2229] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:16.000622 712 raft_consensus.cc:2258] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:16.002589 712 ts_tablet_manager.cc:1875] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:16.015141 562 raft_consensus.cc:1050] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d to VOTER
I20240503 08:13:16.014922 30672 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:16.015861 28354 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:16.016148 562 consensus_queue.cc:237] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:16.028400 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0caf86850b3841ea811b0a6ab19ae330 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:16.032104 642 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:16.036715 30523 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:16.043797 736 tablet_replica.cc:330] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a: stopping tablet replica
I20240503 08:13:16.044239 736 raft_consensus.cc:2229] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:16.044770 736 raft_consensus.cc:2258] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:16.047256 28630 consensus_queue.cc:237] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:16.052268 30281 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:16.053329 683 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:13:16.058871 28570 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d44335d8be7245b1bb52b2eab9e42c18. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:16.063467 28373 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:16.067626 678 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240503 08:13:16.069535 30542 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:16.071424 736 ts_tablet_manager.cc:1875] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:16.076959 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:16.077713 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:16.089738 678 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.090238 28373 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.097890 28227 catalog_manager.cc:5484] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.107888 30676 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:16.108831 32682 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:16.112210 30676 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:16.167774 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:16.186750 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:16.197074 420 raft_consensus.cc:1050] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d to VOTER
I20240503 08:13:16.198031 420 consensus_queue.cc:237] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:16.200459 735 tablet_replica.cc:330] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c: stopping tablet replica
I20240503 08:13:16.200834 30672 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:16.200825 735 raft_consensus.cc:2229] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:16.201520 735 raft_consensus.cc:2258] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:16.202389 542 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:16.202569 30281 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:16.203576 542 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:16.205617 30543 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:16.207667 736 ts_tablet_manager.cc:1888] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 08:13:16.207912 736 log.cc:1196] T 0caf86850b3841ea811b0a6ab19ae330 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/0caf86850b3841ea811b0a6ab19ae330
I20240503 08:13:16.208894 542 raft_consensus.cc:1050] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER c8375219addc48fa99ac474ed1c9aef9 to VOTER
I20240503 08:13:16.210304 542 consensus_queue.cc:237] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:16.211763 28218 catalog_manager.cc:4830] TS 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): tablet 0caf86850b3841ea811b0a6ab19ae330 (table rebalance_test_table_1 [id=b9e797909b184d0bb18e1af0519e9966]) successfully deleted
I20240503 08:13:16.216496 30281 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:16.222618 32682 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:16.214485 675 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:16.225724 30543 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:16.230124 30672 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:16.239611 28231 catalog_manager.cc:5484] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d reported cstate change: config changed from index 10 to 11, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:16.251861 30672 raft_consensus.cc:1261] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:16.252729 542 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:16.255841 30281 raft_consensus.cc:1261] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:16.256805 675 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:16.266419 30676 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:16.267222 739 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:16.268546 28373 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:16.270501 683 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:16.270923 30544 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:16.272830 739 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:16.281467 28501 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:16.312697 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:16.327708 735 ts_tablet_manager.cc:1875] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:16.328058 28762 raft_consensus.cc:1261] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:16.328685 675 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:13:16.343380 731 log.cc:924] Time spent T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20240503 08:13:16.344071 28502 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:16.344997 683 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:16.346915 28502 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
W20240503 08:13:16.350852 28309 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.135:45937, user_credentials={real_user=slave}} blocked reactor thread for 77300.2us
I20240503 08:13:16.359130 28217 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d44335d8be7245b1bb52b2eab9e42c18 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:13:16.364938 30278 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.369587 675 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:16.374284 30281 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:16.374224 28761 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:16.380633 744 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:16.386159 30541 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:16.394053 28630 consensus_queue.cc:237] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:16.397272 30278 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:16.397522 28374 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:16.398257 683 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:16.398743 683 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:13:16.400862 28570 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d44335d8be7245b1bb52b2eab9e42c18. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:16.402025 683 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.402208 28374 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.402983 30278 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.407368 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d44335d8be7245b1bb52b2eab9e42c18 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:16.408913 28232 catalog_manager.cc:5484] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 reported cstate change: config changed from index 7 to 8, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:16.408941 28231 catalog_manager.cc:5484] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 7 to 8, VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) evicted. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.412575 30676 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:16.413806 28484 tablet_service.cc:1514] Processing DeleteTablet for tablet d44335d8be7245b1bb52b2eab9e42c18 with delete_type TABLET_DATA_TOMBSTONED (TS 716ffbb2a00b4f43b03e5cedcc9d302a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:54162
I20240503 08:13:16.414184 736 tablet_replica.cc:330] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a: stopping tablet replica
I20240503 08:13:16.414384 736 raft_consensus.cc:2229] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:16.414707 736 raft_consensus.cc:2258] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:16.415716 28226 catalog_manager.cc:5484] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 7 to 8, c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:16.417337 30672 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:16.423166 712 ts_tablet_manager.cc:1888] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 08:13:16.423344 712 log.cc:1196] T e9997e8da99c4f45a63bc3db521b33ba P b29361eb4afa4ab3af7d8019d8a5042d: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-7/wal/wals/e9997e8da99c4f45a63bc3db521b33ba
I20240503 08:13:16.424396 28218 catalog_manager.cc:4830] TS b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): tablet e9997e8da99c4f45a63bc3db521b33ba (table rebalance_test_table_2 [id=2ff6aeab7e1847f6884750db73f8c428]) successfully deleted
I20240503 08:13:16.434734 736 ts_tablet_manager.cc:1875] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:16.439636 28630 consensus_queue.cc:237] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:16.444067 28502 consensus_queue.cc:237] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:13:16.444715 736 ts_tablet_manager.cc:1888] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:13:16.444845 736 log.cc:1196] T d44335d8be7245b1bb52b2eab9e42c18 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-1/wal/wals/d44335d8be7245b1bb52b2eab9e42c18
I20240503 08:13:16.447378 28218 catalog_manager.cc:4830] TS 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): tablet d44335d8be7245b1bb52b2eab9e42c18 (table rebalance_test_table_5 [id=311fe8a0590b44a8a07f0203d8f65e32]) successfully deleted
I20240503 08:13:16.448098 30542 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:16.448832 562 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:16.451961 744 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:16.458164 30541 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:16.463059 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b4ceeeb3495e4a8fac6e13e431c64af4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:16.465817 28226 catalog_manager.cc:5484] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 8 to 9, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:16.472965 28761 consensus_queue.cc:237] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:13:16.482151 30278 raft_consensus.cc:1261] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:16.483235 675 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:16.487059 675 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:16.489655 30280 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:16.497217 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac392ff7965a405f96efc777aca4f0bb with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:16.500979 28232 catalog_manager.cc:5484] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 8 to 9, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:16.507854 28744 tablet_service.cc:1514] Processing DeleteTablet for tablet ac392ff7965a405f96efc777aca4f0bb with delete_type TABLET_DATA_TOMBSTONED (TS 93f7e733cec9415eb422b0a7ba8621cb not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39918
I20240503 08:13:16.526831 28353 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ceeeb3495e4a8fac6e13e431c64af4 with delete_type TABLET_DATA_TOMBSTONED (TS 4e4a1242f3714d03a6c9ee51dd63929c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51022
I20240503 08:13:16.539052 30674 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:16.540009 678 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:16.547391 30674 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:16.559252 753 tablet_replica.cc:330] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb: stopping tablet replica
I20240503 08:13:16.559515 753 raft_consensus.cc:2229] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:16.559763 753 raft_consensus.cc:2258] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:16.563352 30673 raft_consensus.cc:1261] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:16.566998 675 consensus_queue.cc:1035] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:16.567674 735 ts_tablet_manager.cc:1888] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:13:16.567835 735 log.cc:1196] T a20489bf65774902af38d228ee966ffe P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/a20489bf65774902af38d228ee966ffe
I20240503 08:13:16.568924 30673 raft_consensus.cc:2932] T ac392ff7965a405f96efc777aca4f0bb P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:13:16.570772 753 ts_tablet_manager.cc:1875] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:16.573452 735 tablet_replica.cc:330] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c: stopping tablet replica
I20240503 08:13:16.573663 735 raft_consensus.cc:2229] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:16.574267 28218 catalog_manager.cc:4830] TS 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): tablet a20489bf65774902af38d228ee966ffe (table rebalance_test_table_6 [id=87e8ca2152ac403cb71e5eb67c0b2e0e]) successfully deleted
W20240503 08:13:16.577545 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): Couldn't send request to peer 4574396f35514c9caccb346f1a7af84d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:16.579969 735 pending_rounds.cc:62] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c: Trying to abort 1 pending ops.
I20240503 08:13:16.581110 30672 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:13:16.580112 735 pending_rounds.cc:68] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023509487679434752 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "b4ceeeb3495e4a8fac6e13e431c64af4" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } } }
I20240503 08:13:16.582212 30281 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:13:16.582288 735 raft_consensus.cc:2894] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240503 08:13:16.582442 735 raft_consensus.cc:2866] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:13:16.582598 735 raft_consensus.cc:2258] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:16.583101 676 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:13:16.586398 676 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:13:16.592419 753 ts_tablet_manager.cc:1888] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:16.600993 30672 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:16.607002 735 ts_tablet_manager.cc:1875] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:16.608945 28226 catalog_manager.cc:5484] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d reported cstate change: config changed from index 11 to 12, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:16.623363 30280 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:16.599452 676 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:13:16.627233 753 log.cc:1196] T ac392ff7965a405f96efc777aca4f0bb P 93f7e733cec9415eb422b0a7ba8621cb: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/wals/ac392ff7965a405f96efc777aca4f0bb
I20240503 08:13:16.628190 28218 catalog_manager.cc:4830] TS 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095): tablet ac392ff7965a405f96efc777aca4f0bb (table rebalance_test_table_2 [id=2ff6aeab7e1847f6884750db73f8c428]) successfully deleted
I20240503 08:13:16.649873 735 ts_tablet_manager.cc:1888] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:16.650048 735 log.cc:1196] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/b4ceeeb3495e4a8fac6e13e431c64af4
I20240503 08:13:16.650858 28218 catalog_manager.cc:4830] TS 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): tablet b4ceeeb3495e4a8fac6e13e431c64af4 (table rebalance_test_table_7 [id=0225804a5c4a42cbb32c4b0029752bd1]) successfully deleted
I20240503 08:13:16.731330 28373 consensus_queue.cc:237] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
W20240503 08:13:16.742554 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d13c33b8712e4c488abb99b73ff42b86. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:16.743817 30541 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:16.744436 673 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:16.747224 28630 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:16.747763 673 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:16.751747 673 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:16.753504 30541 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:16.769870 28232 catalog_manager.cc:5484] T d13c33b8712e4c488abb99b73ff42b86 P bea2145b8e24436392b18ec3cfe90b93 reported cstate change: config changed from index 6 to 7, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:16.800396 28630 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:16.869725 787 ts_tablet_manager.cc:902] T d44335d8be7245b1bb52b2eab9e42c18 P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:13:16.878487 28373 consensus_queue.cc:237] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:16.892984 30412 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:16.894049 673 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:16.897516 28630 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:16.898454 673 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:16.900367 28502 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:16.901090 745 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:13:16.901880 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 173376257c4946b5a801a27e4dffe90d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:16.907565 673 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.908962 28630 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.913766 30412 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.925513 28501 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.926930 28232 catalog_manager.cc:5484] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 5 to 6, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.933776 28505 consensus_queue.cc:237] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:16.939213 30541 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:16.940213 675 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:16.956904 28373 consensus_queue.cc:237] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:16.959918 28502 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:16.960737 32559 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:13:16.961660 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 173376257c4946b5a801a27e4dffe90d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:16.963510 32559 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.964188 28502 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.973712 28232 catalog_manager.cc:5484] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 6 to 7, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.980672 28614 tablet_service.cc:1514] Processing DeleteTablet for tablet 173376257c4946b5a801a27e4dffe90d with delete_type TABLET_DATA_TOMBSTONED (TS 4574396f35514c9caccb346f1a7af84d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56340
I20240503 08:13:16.983844 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 173376257c4946b5a801a27e4dffe90d with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:16.985430 28761 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:16.994570 30414 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:16.995637 32559 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:16.997119 815 tablet_replica.cc:330] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d: stopping tablet replica
I20240503 08:13:16.997931 815 raft_consensus.cc:2229] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:16.998270 815 raft_consensus.cc:2258] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:16.998311 30412 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:16.999441 815 ts_tablet_manager.cc:1875] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:17.003826 787 tablet_copy_client.cc:308] T d44335d8be7245b1bb52b2eab9e42c18 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:13:17.003739 420 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:13:17.007622 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5736ee8065344ea8124fdf759035216. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:17.008225 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet d44335d8be7245b1bb52b2eab9e42c18 from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:40357)
I20240503 08:13:17.008390 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet d44335d8be7245b1bb52b2eab9e42c18 from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:40357: session id = bea2145b8e24436392b18ec3cfe90b93-d44335d8be7245b1bb52b2eab9e42c18
I20240503 08:13:17.011206 28655 tablet_copy_source_session.cc:205] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:17.013180 787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44335d8be7245b1bb52b2eab9e42c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:17.022315 787 tablet_copy_client.cc:783] T d44335d8be7245b1bb52b2eab9e42c18 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:17.022809 787 tablet_copy_client.cc:647] T d44335d8be7245b1bb52b2eab9e42c18 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:17.024247 815 ts_tablet_manager.cc:1888] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:13:17.024394 815 log.cc:1196] T 173376257c4946b5a801a27e4dffe90d P 4574396f35514c9caccb346f1a7af84d: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-2/wal/wals/173376257c4946b5a801a27e4dffe90d
I20240503 08:13:17.025154 787 tablet_copy_client.cc:523] T d44335d8be7245b1bb52b2eab9e42c18 P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:17.025251 28217 catalog_manager.cc:4830] TS 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): tablet 173376257c4946b5a801a27e4dffe90d (table rebalance_test_table_4 [id=3a90dbe17357489db92bd286b71aa565]) successfully deleted
I20240503 08:13:17.030032 420 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:17.032931 28761 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:17.045444 28232 catalog_manager.cc:5484] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 8 to 9, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:17.049831 787 tablet_bootstrap.cc:492] T d44335d8be7245b1bb52b2eab9e42c18 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
I20240503 08:13:17.070003 30542 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:13:17.115927 28630 consensus_queue.cc:237] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:17.126976 30676 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:13:17.128196 28570 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09156806b0744752969b6a574e24b50a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:17.129263 819 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:17.147002 28761 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:17.147787 819 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:17.148734 819 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:17.150315 787 tablet_bootstrap.cc:492] T d44335d8be7245b1bb52b2eab9e42c18 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:17.150841 787 tablet_bootstrap.cc:492] T d44335d8be7245b1bb52b2eab9e42c18 P bea2145b8e24436392b18ec3cfe90b93: Bootstrap complete.
I20240503 08:13:17.151176 787 ts_tablet_manager.cc:1367] T d44335d8be7245b1bb52b2eab9e42c18 P bea2145b8e24436392b18ec3cfe90b93: Time spent bootstrapping tablet: real 0.101s user 0.054s sys 0.015s
I20240503 08:13:17.152241 787 raft_consensus.cc:348] T d44335d8be7245b1bb52b2eab9e42c18 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:17.152575 787 raft_consensus.cc:729] T d44335d8be7245b1bb52b2eab9e42c18 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Initialized, Role: LEARNER
I20240503 08:13:17.152842 787 consensus_queue.cc:260] T d44335d8be7245b1bb52b2eab9e42c18 P bea2145b8e24436392b18ec3cfe90b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:17.154829 787 ts_tablet_manager.cc:1398] T d44335d8be7245b1bb52b2eab9e42c18 P bea2145b8e24436392b18ec3cfe90b93: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:13:17.157434 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session bea2145b8e24436392b18ec3cfe90b93-d44335d8be7245b1bb52b2eab9e42c18 received from {username='slave'} at 127.20.107.135:40357
I20240503 08:13:17.157586 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-d44335d8be7245b1bb52b2eab9e42c18 on tablet d44335d8be7245b1bb52b2eab9e42c18 with peer bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:17.163537 30676 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:17.165987 28227 catalog_manager.cc:5484] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:17.168255 28631 consensus_queue.cc:237] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:17.176146 28761 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:17.191821 859 ts_tablet_manager.cc:902] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
W20240503 08:13:17.210059 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d113e8de02f447b9da0a1cb03686cd2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:17.211321 30412 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:17.212050 843 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:17.225405 28761 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:17.226208 843 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:17.229887 819 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:17.230967 28761 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:17.239194 28226 catalog_manager.cc:5484] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 9 to 10, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:17.270494 30414 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:13:17.294476 28630 consensus_queue.cc:237] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:17.305114 28373 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:17.306119 30676 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:13:17.306598 28570 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d60a162155504975a39c20eff88341e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:17.306792 819 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:17.308506 843 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:17.317015 678 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:17.317684 28373 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:17.322275 30542 raft_consensus.cc:1203] T d44335d8be7245b1bb52b2eab9e42c18 P bea2145b8e24436392b18ec3cfe90b93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:13:17.327358 28226 catalog_manager.cc:5484] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:17.332798 859 tablet_copy_client.cc:308] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:13:17.334481 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet c6e1b9f104ea471cab7527bd56433327 from peer 4574396f35514c9caccb346f1a7af84d ({username='slave'} at 127.20.107.131:42119)
I20240503 08:13:17.334640 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet c6e1b9f104ea471cab7527bd56433327 from peer 4574396f35514c9caccb346f1a7af84d at {username='slave'} at 127.20.107.131:42119: session id = 4574396f35514c9caccb346f1a7af84d-c6e1b9f104ea471cab7527bd56433327
I20240503 08:13:17.337604 28785 tablet_copy_source_session.cc:205] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:17.340466 30676 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } } }
I20240503 08:13:17.349931 859 ts_tablet_manager.cc:1875] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:13:17.355731 888 ts_tablet_manager.cc:902] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:17.369657 859 ts_tablet_manager.cc:1888] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.10
I20240503 08:13:17.370244 859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e1b9f104ea471cab7527bd56433327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:17.374367 859 tablet_copy_client.cc:783] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:17.374835 859 tablet_copy_client.cc:647] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:17.376979 859 tablet_copy_client.cc:523] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:17.382158 859 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:13:17.418813 888 tablet_copy_client.cc:308] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:17.419955 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet d13c33b8712e4c488abb99b73ff42b86 from peer 21d21a5b596b4d48a17a1fb54a972844 ({username='slave'} at 127.20.107.134:32851)
I20240503 08:13:17.420248 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet d13c33b8712e4c488abb99b73ff42b86 from peer 21d21a5b596b4d48a17a1fb54a972844 at {username='slave'} at 127.20.107.134:32851: session id = 21d21a5b596b4d48a17a1fb54a972844-d13c33b8712e4c488abb99b73ff42b86
I20240503 08:13:17.426162 28394 tablet_copy_source_session.cc:205] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:17.431761 888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d13c33b8712e4c488abb99b73ff42b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:17.444420 787 ts_tablet_manager.cc:902] T 173376257c4946b5a801a27e4dffe90d P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:13:17.452826 888 tablet_copy_client.cc:783] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:17.453164 888 tablet_copy_client.cc:647] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:17.455390 888 tablet_copy_client.cc:523] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:17.460481 888 tablet_bootstrap.cc:492] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap starting.
I20240503 08:13:17.506755 787 tablet_copy_client.cc:308] T 173376257c4946b5a801a27e4dffe90d P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:13:17.507544 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet 173376257c4946b5a801a27e4dffe90d from peer bea2145b8e24436392b18ec3cfe90b93 ({username='slave'} at 127.20.107.135:40941)
I20240503 08:13:17.507790 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet 173376257c4946b5a801a27e4dffe90d from peer bea2145b8e24436392b18ec3cfe90b93 at {username='slave'} at 127.20.107.135:40941: session id = bea2145b8e24436392b18ec3cfe90b93-173376257c4946b5a801a27e4dffe90d
I20240503 08:13:17.511073 28394 tablet_copy_source_session.cc:205] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:17.512498 787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 173376257c4946b5a801a27e4dffe90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:17.522150 918 ts_tablet_manager.cc:902] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:13:17.534879 787 tablet_copy_client.cc:783] T 173376257c4946b5a801a27e4dffe90d P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:17.535511 787 tablet_copy_client.cc:647] T 173376257c4946b5a801a27e4dffe90d P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:17.541949 787 tablet_copy_client.cc:523] T 173376257c4946b5a801a27e4dffe90d P bea2145b8e24436392b18ec3cfe90b93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:17.559592 787 tablet_bootstrap.cc:492] T 173376257c4946b5a801a27e4dffe90d P bea2145b8e24436392b18ec3cfe90b93: Bootstrap starting.
I20240503 08:13:17.569020 859 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:17.569569 859 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: Bootstrap complete.
I20240503 08:13:17.569921 859 ts_tablet_manager.cc:1367] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.188s user 0.078s sys 0.017s
I20240503 08:13:17.570884 859 raft_consensus.cc:348] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:13:17.571229 859 raft_consensus.cc:729] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: LEARNER
I20240503 08:13:17.571492 859 consensus_queue.cc:260] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:13:17.579598 859 ts_tablet_manager.cc:1398] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240503 08:13:17.580332 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session 4574396f35514c9caccb346f1a7af84d-c6e1b9f104ea471cab7527bd56433327 received from {username='slave'} at 127.20.107.131:42119
I20240503 08:13:17.580509 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session 4574396f35514c9caccb346f1a7af84d-c6e1b9f104ea471cab7527bd56433327 on tablet c6e1b9f104ea471cab7527bd56433327 with peer 4574396f35514c9caccb346f1a7af84d
I20240503 08:13:17.653096 918 tablet_copy_client.cc:308] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
W20240503 08:13:17.702315 28308 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.134:34587, user_credentials={real_user=slave}} blocked reactor thread for 60805.7us
I20240503 08:13:17.703887 960 ts_tablet_manager.cc:902] T 09156806b0744752969b6a574e24b50a P bea2145b8e24436392b18ec3cfe90b93: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
F20240503 08:13:17.706441 960 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-09156806b0744752969b6a574e24b50a.queue_time_us, tablet-download-pool-09156806b0744752969b6a574e24b50a.queue_time_us) (tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.queue_time_us, tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.queue_time_us) (tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.run_cpu_time_us, tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.run_cpu_time_us) (tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.run_wall_time_us, tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.run_wall_time_us) (tablet-download-pool-173376257c4946b5a801a27e4dffe90d.queue_time_us, tablet-download-pool-173376257c4946b5a801a27e4dffe90d.queue_time_us) (tablet-download-pool-173376257c4946b5a801a27e4dffe90d.run_cpu_time_us, tablet-download-pool-173376257c4946b5a801a27e4dffe90d.run_cpu_time_us) (tablet-download-pool-173376257c4946b5a801a27e4dffe90d.run_wall_time_us, tablet-download-pool-173376257c4946b5a801a27e4dffe90d.run_wall_time_us) (tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.queue_time_us, tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.queue_time_us) (tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.run_cpu_time_us, tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.run_cpu_time_us) (tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.run_wall_time_us, tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.run_wall_time_us) (tablet-download-pool-48e31346107a453a81921b23b244e7ff.queue_time_us, tablet-download-pool-48e31346107a453a81921b23b244e7ff.queue_time_us) (tablet-download-pool-48e31346107a453a81921b23b244e7ff.run_cpu_time_us, tablet-download-pool-48e31346107a453a81921b23b244e7ff.run_cpu_time_us) (tablet-download-pool-48e31346107a453a81921b23b244e7ff.run_wall_time_us, tablet-download-pool-48e31346107a453a81921b23b244e7ff.run_wall_time_us) (tablet-download-pool-4babf0ea9b954a23b1b1c4e403dce0bd.queue_time_us, tablet-download-pool-4babf0ea9b954a23b1b1c4e403dce0bd.queue_time_us) (tablet-download-pool-4babf0ea9b954a23b1b1c4e403dce0bd.run_cpu_time_us, tablet-download-pool-4babf0ea9b954a23b1b1c4e403dce0bd.run_cpu_time_us) (tablet-download-pool-4babf0ea9b954a23b1b1c4e403dce0bd.run_wall_time_us, tablet-download-pool-4babf0ea9b954a23b1b1c4e403dce0bd.run_wall_time_us) (tablet-download-pool-a20489bf65774902af38d228ee966ffe.queue_time_us, tablet-download-pool-a20489bf65774902af38d228ee966ffe.queue_time_us) (tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_cpu_time_us, tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_cpu_time_us) (tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_wall_time_us, tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_wall_time_us) (tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.queue_time_us, tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.queue_time_us) (tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.run_cpu_time_us, tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.run_cpu_time_us) (tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.run_wall_time_us, tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.run_wall_time_us) (tablet-download-pool-b4ceeeb3495e4a8fac6e13e431c64af4.queue_time_us, tablet-download-pool-b4ceeeb3495e4a8fac6e13e431c64af4.queue_time_us) (tablet-download-pool-b4ceeeb3495e4a8fac6e13e431c64af4.run_cpu_time_us, tablet-download-pool-b4ceeeb3495e4a8fac6e13e431c64af4.run_cpu_time_us) (tablet-download-pool-b4ceeeb3495e4a8fac6e13e431c64af4.run_wall_time_us, tablet-download-pool-b4ceeeb3495e4a8fac6e13e431c64af4.run_wall_time_us) (tablet-download-pool-b5736ee8065344ea8124fdf759035216.queue_time_us, tablet-download-pool-b5736ee8065344ea8124fdf759035216.queue_time_us) (tablet-download-pool-b5736ee8065344ea8124fdf759035216.run_cpu_time_us, tablet-download-pool-b5736ee8065344ea8124fdf759035216.run_cpu_time_us) (tablet-download-pool-b5736ee8065344ea8124fdf759035216.run_wall_time_us, tablet-download-pool-b5736ee8065344ea8124fdf759035216.run_wall_time_us) (tablet-download-pool-ba833ce23c4c4e81bc1a81125a1208c9.queue_time_us, tablet-download-pool-ba833ce23c4c4e81bc1a81125a1208c9.queue_time_us) (tablet-download-pool-ba833ce23c4c4e81bc1a81125a1208c9.run_cpu_time_us, tablet-download-pool-ba833ce23c4c4e81bc1a81125a1208c9.run_cpu_time_us) (tablet-download-pool-ba833ce23c4c4e81bc1a81125a1208c9.run_wall_time_us, tablet-download-pool-ba833ce23c4c4e81bc1a81125a1208c9.run_wall_time_us) (tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.queue_time_us, tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.queue_time_us) (tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.run_cpu_time_us, tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.run_cpu_time_us) (tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.run_wall_time_us, tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.run_wall_time_us) (tablet-download-pool-d13c33b8712e4c488abb99b73ff42b86.queue_time_us, tablet-download-pool-d13c33b8712e4c488abb99b73ff42b86.queue_time_us) (tablet-download-pool-d13c33b8712e4c488abb99b73ff42b86.run_cpu_time_us, tablet-download-pool-d13c33b8712e4c488abb99b73ff42b86.run_cpu_time_us) (tablet-download-pool-d13c33b8712e4c488abb99b73ff42b86.run_wall_time_us, tablet-download-pool-d13c33b8712e4c488abb99b73ff42b86.run_wall_time_us) (tablet-download-pool-d44335d8be7245b1bb52b2eab9e42c18.queue_time_us, tablet-download-pool-d44335d8be7245b1bb52b2eab9e42c18.queue_time_us) (tablet-download-pool-d44335d8be7245b1bb52b2eab9e42c18.run_cpu_time_us, tablet-download-pool-d44335d8be7245b1bb52b2eab9e42c18.run_cpu_time_us) (tablet-download-pool-d44335d8be7245b1bb52b2eab9e42c18.run_wall_time_us, tablet-download-pool-d44335d8be7245b1bb52b2eab9e42c18.run_wall_time_us) (tablet-download-pool-ede24a27608a4cf196dd46f85c4163e1.queue_time_us, tablet-download-pool-ede24a27608a4cf196dd46f85c4163e1.queue_time_us) (tablet-download-pool-ede24a27608a4cf196dd46f85c4163e1.run_cpu_time_us, tablet-download-pool-ede24a27608a4cf196dd46f85c4163e1.run_cpu_time_us) (tablet-download-pool-ede24a27608a4cf196dd46f85c4163e1.run_wall_time_us, tablet-download-pool-ede24a27608a4cf196dd46f85c4163e1.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 08:13:17.709897 28630 raft_consensus.cc:1203] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
@ 0x7ffbbd37cdcd google::LogMessage::Fail() at ??:0
@ 0x7ffbbd380b93 google::LogMessage::SendToLog() at ??:0
I20240503 08:13:17.716121 28524 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet b5736ee8065344ea8124fdf759035216 from peer 21d21a5b596b4d48a17a1fb54a972844 ({username='slave'} at 127.20.107.134:45771)
I20240503 08:13:17.716324 28524 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet b5736ee8065344ea8124fdf759035216 from peer 21d21a5b596b4d48a17a1fb54a972844 at {username='slave'} at 127.20.107.134:45771: session id = 21d21a5b596b4d48a17a1fb54a972844-b5736ee8065344ea8124fdf759035216
I20240503 08:13:17.719255 28524 tablet_copy_source_session.cc:205] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:17.722363 918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5736ee8065344ea8124fdf759035216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:17.726203 28373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ba833ce23c4c4e81bc1a81125a1208c9"
dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43194
I20240503 08:13:17.726514 28373 raft_consensus.cc:595] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:17.730710 918 tablet_copy_client.cc:783] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 0 data blocks...
@ 0x7ffbbd37c7cc google::LogMessage::Flush() at ??:0
@ 0x7ffbbd37df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 08:13:17.746693 918 tablet_copy_client.cc:647] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:17.749578 918 tablet_copy_client.cc:523] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7ffbc00bd742 kudu::TraceMetrics::InternName() at ??:0
I20240503 08:13:17.751642 955 raft_consensus.cc:979] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c: : Instructing follower bea2145b8e24436392b18ec3cfe90b93 to start an election
I20240503 08:13:17.751828 955 raft_consensus.cc:1067] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Signalling peer bea2145b8e24436392b18ec3cfe90b93 to start an election
I20240503 08:13:17.753099 30542 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ba833ce23c4c4e81bc1a81125a1208c9"
dest_uuid: "bea2145b8e24436392b18ec3cfe90b93"
from {username='slave'} at 127.20.107.129:45317
I20240503 08:13:17.753417 30542 raft_consensus.cc:482] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:17.753656 30542 raft_consensus.cc:3037] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:17.753912 918 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap starting.
I20240503 08:13:17.757071 30542 raft_consensus.cc:504] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
@ 0x7ffbc0089f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240503 08:13:17.760181 30542 leader_election.cc:288] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93 [CANDIDATE]: Term 2 election: Requested vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:13:17.760938 28630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba833ce23c4c4e81bc1a81125a1208c9" candidate_uuid: "bea2145b8e24436392b18ec3cfe90b93" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:13:17.761186 28630 raft_consensus.cc:3037] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Advancing to term 2
@ 0x7ffbc0083cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 08:13:17.764425 28630 raft_consensus.cc:2454] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bea2145b8e24436392b18ec3cfe90b93 in term 2.
I20240503 08:13:17.765075 28502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba833ce23c4c4e81bc1a81125a1208c9" candidate_uuid: "bea2145b8e24436392b18ec3cfe90b93" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:13:17.765321 28502 raft_consensus.cc:3037] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:17.768239 28502 raft_consensus.cc:2454] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bea2145b8e24436392b18ec3cfe90b93 in term 2.
@ 0x7ffbcf4d78de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 08:13:17.769130 30477 leader_election.cc:302] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 716ffbb2a00b4f43b03e5cedcc9d302a, bea2145b8e24436392b18ec3cfe90b93; no voters:
I20240503 08:13:17.769596 725 raft_consensus.cc:2781] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:17.770051 28373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba833ce23c4c4e81bc1a81125a1208c9" candidate_uuid: "bea2145b8e24436392b18ec3cfe90b93" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:13:17.770295 28373 raft_consensus.cc:3032] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:17.770390 28373 raft_consensus.cc:729] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Running, Role: LEADER
I20240503 08:13:17.770776 28373 consensus_queue.cc:260] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:13:17.771292 28373 raft_consensus.cc:3037] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Advancing to term 2
@ 0x7ffbcf4efdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7ffbcf61a480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 08:13:17.786278 28373 raft_consensus.cc:2454] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bea2145b8e24436392b18ec3cfe90b93 in term 2.
@ 0x7ffbcf652a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 08:13:17.794459 978 ts_tablet_manager.cc:902] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
@ 0x7ffbcf635981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7ffbcf63570e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ffbc008db75 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 08:13:17.811596 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7ffbc009119c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ffbc0091019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ffbc006c01e kudu::Thread::SuperviseThread() at ??:0
I20240503 08:13:17.823715 725 raft_consensus.cc:686] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93 [term 2 LEADER]: Becoming Leader. State: Replica: bea2145b8e24436392b18ec3cfe90b93, State: Running, Role: LEADER
I20240503 08:13:17.824467 725 consensus_queue.cc:237] T ba833ce23c4c4e81bc1a81125a1208c9 P bea2145b8e24436392b18ec3cfe90b93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
@ 0x7ffbc33f16db start_thread at ??:0
@ 0x7ffbbc22f71f clone at ??:0
I20240503 08:13:17.837710 28353 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:17.879061 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:17.900581 888 tablet_bootstrap.cc:492] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:17.900846 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:17.903048 888 tablet_bootstrap.cc:492] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap complete.
I20240503 08:13:17.906690 888 ts_tablet_manager.cc:1367] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844: Time spent bootstrapping tablet: real 0.443s user 0.049s sys 0.010s
I20240503 08:13:17.923364 978 tablet_copy_client.cc:308] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:13:17.924258 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet 2d113e8de02f447b9da0a1cb03686cd2 from peer b29361eb4afa4ab3af7d8019d8a5042d ({username='slave'} at 127.20.107.136:48299)
I20240503 08:13:17.924419 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet 2d113e8de02f447b9da0a1cb03686cd2 from peer b29361eb4afa4ab3af7d8019d8a5042d at {username='slave'} at 127.20.107.136:48299: session id = b29361eb4afa4ab3af7d8019d8a5042d-2d113e8de02f447b9da0a1cb03686cd2
I20240503 08:13:17.928256 28655 tablet_copy_source_session.cc:205] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:17.930646 978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d113e8de02f447b9da0a1cb03686cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:17.980757 978 tablet_copy_client.cc:783] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:17.990285 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:17.991742 978 tablet_copy_client.cc:647] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:17.999655 978 tablet_copy_client.cc:523] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:18.020138 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:18.020478 888 raft_consensus.cc:348] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:18.020753 888 raft_consensus.cc:729] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Initialized, Role: LEARNER
I20240503 08:13:18.020999 888 consensus_queue.cc:260] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:18.044728 978 tablet_bootstrap.cc:492] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap starting.
I20240503 08:13:18.047376 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:18.048264 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.055424 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.056728 888 ts_tablet_manager.cc:1398] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844: Time spent starting tablet: real 0.150s user 0.004s sys 0.000s
I20240503 08:13:18.065012 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session 21d21a5b596b4d48a17a1fb54a972844-d13c33b8712e4c488abb99b73ff42b86 received from {username='slave'} at 127.20.107.134:32851
I20240503 08:13:18.065217 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session 21d21a5b596b4d48a17a1fb54a972844-d13c33b8712e4c488abb99b73ff42b86 on tablet d13c33b8712e4c488abb99b73ff42b86 with peer 21d21a5b596b4d48a17a1fb54a972844
I20240503 08:13:18.097790 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:18.113636 28353 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:18.118922 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:18.125614 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.172617 918 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:18.201561 918 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap complete.
I20240503 08:13:18.201910 918 ts_tablet_manager.cc:1367] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844: Time spent bootstrapping tablet: real 0.448s user 0.057s sys 0.003s
I20240503 08:13:18.202811 918 raft_consensus.cc:348] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:18.203089 918 raft_consensus.cc:729] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Initialized, Role: LEARNER
I20240503 08:13:18.203277 918 consensus_queue.cc:260] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:18.203891 918 ts_tablet_manager.cc:1398] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:18.215313 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.220075 28524 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session 21d21a5b596b4d48a17a1fb54a972844-b5736ee8065344ea8124fdf759035216 received from {username='slave'} at 127.20.107.134:45771
I20240503 08:13:18.220242 28524 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session 21d21a5b596b4d48a17a1fb54a972844-b5736ee8065344ea8124fdf759035216 on tablet b5736ee8065344ea8124fdf759035216 with peer 21d21a5b596b4d48a17a1fb54a972844
I20240503 08:13:18.352453 1052 raft_consensus.cc:1050] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER 4574396f35514c9caccb346f1a7af84d to VOTER
I20240503 08:13:18.353960 1052 consensus_queue.cc:237] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } }
I20240503 08:13:18.411804 28631 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:13:18.424861 1066 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20240503 08:13:18.426331 30412 raft_consensus.cc:1203] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240503 08:13:18.433503 28700 connection.cc:535] client connection to 127.20.107.135:45937 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:18.433838 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:18.434826 28440 connection.cc:535] client connection to 127.20.107.135:45937 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:18.435074 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:18.435567 28309 connection.cc:535] client connection to 127.20.107.135:45937 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:18.435876 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:18.445302 28570 connection.cc:535] client connection to 127.20.107.135:45937 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:18.445621 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:18.460024 28698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.131:44559, user_credentials={real_user=slave}} blocked reactor thread for 47708.4us
I20240503 08:13:18.461048 30280 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:13:18.473031 1066 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 08:13:18.476038 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:18.476199 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:18.476289 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:18.476389 28570 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:18.476821 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:18.476995 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:18.477093 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:18.491218 28699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.107.129:38529, user_credentials={real_user=slave}} blocked reactor thread for 103334us
I20240503 08:13:18.492144 30672 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:13:18.495858 676 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:13:18.506435 676 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:18.513269 28631 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:18.513830 30672 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:18.517305 30280 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:13:18.525117 28231 catalog_manager.cc:5484] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 12 to 13, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
W20240503 08:13:18.529145 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:18.558936 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:18.559123 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.588461 30412 raft_consensus.cc:1203] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 08:13:18.592612 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.657470 30394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.657976 28353 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:18.659138 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:18.672765 28614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:18.690886 28374 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b3b8b80ebd44f27899f97211266c7b2"
dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43226
I20240503 08:13:18.691140 28374 raft_consensus.cc:595] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:18.692895 745 raft_consensus.cc:979] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c: : Instructing follower 21d21a5b596b4d48a17a1fb54a972844 to start an election
I20240503 08:13:18.693063 745 raft_consensus.cc:1067] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Signalling peer 21d21a5b596b4d48a17a1fb54a972844 to start an election
I20240503 08:13:18.694027 30414 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b3b8b80ebd44f27899f97211266c7b2"
dest_uuid: "21d21a5b596b4d48a17a1fb54a972844"
from {username='slave'} at 127.20.107.129:33387
I20240503 08:13:18.694304 30414 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:18.694515 30414 raft_consensus.cc:3037] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:18.698419 30414 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:18.707077 28353 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:18.722496 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.736677 30414 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [CANDIDATE]: Term 2 election: Requested vote from peers 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095), 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389)
I20240503 08:13:18.743964 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3b8b80ebd44f27899f97211266c7b2" candidate_uuid: "21d21a5b596b4d48a17a1fb54a972844" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:13:18.744235 28374 raft_consensus.cc:3032] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:18.744349 28374 raft_consensus.cc:729] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Running, Role: LEADER
I20240503 08:13:18.744665 28374 consensus_queue.cc:260] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:18.745153 28374 raft_consensus.cc:3037] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:18.749310 28374 raft_consensus.cc:2454] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d21a5b596b4d48a17a1fb54a972844 in term 2.
W20240503 08:13:18.767170 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:13:18.776422 28761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3b8b80ebd44f27899f97211266c7b2" candidate_uuid: "21d21a5b596b4d48a17a1fb54a972844" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "93f7e733cec9415eb422b0a7ba8621cb"
I20240503 08:13:18.776705 28761 raft_consensus.cc:3037] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:18.779872 28761 raft_consensus.cc:2454] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d21a5b596b4d48a17a1fb54a972844 in term 2.
W20240503 08:13:18.781481 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:13:18.790050 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:18.802539 30347 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 21d21a5b596b4d48a17a1fb54a972844, 4e4a1242f3714d03a6c9ee51dd63929c, 93f7e733cec9415eb422b0a7ba8621cb; no voters:
I20240503 08:13:18.802950 1028 raft_consensus.cc:2781] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:18.807410 28744 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:18.825546 30261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.834043 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:18.852233 1028 raft_consensus.cc:686] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [term 2 LEADER]: Becoming Leader. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Running, Role: LEADER
I20240503 08:13:18.852852 1028 consensus_queue.cc:237] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:18.857231 28484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:18.865516 30676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3b8b80ebd44f27899f97211266c7b2" candidate_uuid: "21d21a5b596b4d48a17a1fb54a972844" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b29361eb4afa4ab3af7d8019d8a5042d"
I20240503 08:13:18.865888 30676 raft_consensus.cc:3037] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:18.869665 30676 raft_consensus.cc:2454] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d21a5b596b4d48a17a1fb54a972844 in term 2.
I20240503 08:13:18.887712 28232 catalog_manager.cc:5484] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: term changed from 1 to 2, leader changed from 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) to 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134). New cstate: current_term: 2 leader_uuid: "21d21a5b596b4d48a17a1fb54a972844" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:18.897192 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.937141 978 tablet_bootstrap.cc:492] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:18.940472 978 tablet_bootstrap.cc:492] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d: Bootstrap complete.
I20240503 08:13:18.940789 978 ts_tablet_manager.cc:1367] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent bootstrapping tablet: real 0.896s user 0.050s sys 0.013s
I20240503 08:13:18.942663 978 raft_consensus.cc:348] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:18.942942 978 raft_consensus.cc:729] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b29361eb4afa4ab3af7d8019d8a5042d, State: Initialized, Role: LEARNER
I20240503 08:13:18.943136 978 consensus_queue.cc:260] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:18.954366 30656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.954496 978 ts_tablet_manager.cc:1398] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 08:13:18.985170 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-2d113e8de02f447b9da0a1cb03686cd2 received from {username='slave'} at 127.20.107.136:48299
I20240503 08:13:18.985358 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session b29361eb4afa4ab3af7d8019d8a5042d-2d113e8de02f447b9da0a1cb03686cd2 on tablet 2d113e8de02f447b9da0a1cb03686cd2 with peer b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:18.992610 955 raft_consensus.cc:1050] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 to VOTER
I20240503 08:13:18.993572 955 consensus_queue.cc:237] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:19.010015 30414 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:19.010665 745 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:13:19.019330 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:19.020740 28631 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:19.021871 745 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:19.106906 28630 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d44335d8be7245b1bb52b2eab9e42c18"
dest_uuid: "4574396f35514c9caccb346f1a7af84d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41026
I20240503 08:13:19.107250 28630 raft_consensus.cc:595] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:19.112008 955 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:19.117177 30414 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:19.121380 28631 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:19.125850 28228 catalog_manager.cc:5484] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 7 to 8, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:19.147253 1031 raft_consensus.cc:1050] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 to VOTER
I20240503 08:13:19.148382 1031 consensus_queue.cc:237] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:19.185740 30414 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:19.186424 1032 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 08:13:19.191538 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:19.192960 28761 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:13:19.193742 1032 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:19.236565 1032 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:19.239688 28761 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Refusing update from remote peer 21d21a5b596b4d48a17a1fb54a972844: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:19.240372 1028 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:19.241983 30412 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:19.249922 28226 catalog_manager.cc:5484] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 9 to 10, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:19.252293 28761 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
W20240503 08:13:19.274755 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:19.338603 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:13:19.343734 30676 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 2 FOLLOWER]: Refusing update from remote peer 21d21a5b596b4d48a17a1fb54a972844: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:19.344489 1028 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:19.359807 30676 raft_consensus.cc:1203] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 08:13:19.374970 28374 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 2 FOLLOWER]: Refusing update from remote peer 21d21a5b596b4d48a17a1fb54a972844: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:19.375644 1028 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:19.379011 30414 consensus_queue.cc:237] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:13:19.416849 30676 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 2 FOLLOWER]: Refusing update from remote peer 21d21a5b596b4d48a17a1fb54a972844: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:19.417917 1184 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:19.419308 28761 raft_consensus.cc:1261] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Refusing update from remote peer 21d21a5b596b4d48a17a1fb54a972844: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:19.422483 1184 consensus_queue.cc:1035] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:19.426899 1139 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:19.427629 28761 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:19.440888 30672 raft_consensus.cc:2932] T 9b3b8b80ebd44f27899f97211266c7b2 P b29361eb4afa4ab3af7d8019d8a5042d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:19.450512 1048 raft_consensus.cc:979] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d: : Instructing follower 4e4a1242f3714d03a6c9ee51dd63929c to start an election
I20240503 08:13:19.431684 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b3b8b80ebd44f27899f97211266c7b2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 08:13:19.584728 28218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.20.107.134:34587, user_credentials={real_user=slave}} blocked reactor thread for 153010us
I20240503 08:13:19.567507 843 raft_consensus.cc:482] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:19.585693 843 raft_consensus.cc:504] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:13:19.586680 843 leader_election.cc:288] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397), bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937)
I20240503 08:13:19.586879 843 raft_consensus.cc:1067] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Signalling peer 4e4a1242f3714d03a6c9ee51dd63929c to start an election
I20240503 08:13:19.454453 28226 catalog_manager.cc:5484] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 8 to 10, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New cstate: current_term: 2 leader_uuid: "21d21a5b596b4d48a17a1fb54a972844" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:13:19.591706 28502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba833ce23c4c4e81bc1a81125a1208c9" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" is_pre_election: true
I20240503 08:13:19.591943 28502 raft_consensus.cc:2454] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 2.
I20240503 08:13:19.595331 28374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba833ce23c4c4e81bc1a81125a1208c9" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" is_pre_election: true
I20240503 08:13:19.595599 28374 raft_consensus.cc:2454] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 2.
I20240503 08:13:19.595548 28373 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d44335d8be7245b1bb52b2eab9e42c18"
dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
from {username='slave'} at 127.20.107.131:46685
I20240503 08:13:19.595805 28373 raft_consensus.cc:482] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:19.595913 28373 raft_consensus.cc:3037] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:19.599092 28373 raft_consensus.cc:504] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:19.600809 28569 leader_election.cc:302] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:13:19.601557 28631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44335d8be7245b1bb52b2eab9e42c18" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4574396f35514c9caccb346f1a7af84d"
I20240503 08:13:19.601779 28631 raft_consensus.cc:3032] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:19.601861 28631 raft_consensus.cc:729] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Running, Role: LEADER
I20240503 08:13:19.602195 28631 consensus_queue.cc:260] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:19.602619 28631 raft_consensus.cc:3037] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:19.605201 30280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44335d8be7245b1bb52b2eab9e42c18" candidate_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "c8375219addc48fa99ac474ed1c9aef9"
I20240503 08:13:19.605557 30280 raft_consensus.cc:3037] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:13:19.607565 28570 leader_election.cc:334] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:13:19.608546 30280 raft_consensus.cc:2454] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 2.
I20240503 08:13:19.599957 28373 leader_election.cc:288] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 2 election: Requested vote from peers 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559), c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121)
I20240503 08:13:19.606987 28353 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b3b8b80ebd44f27899f97211266c7b2 with delete_type TABLET_DATA_TOMBSTONED (TS 4e4a1242f3714d03a6c9ee51dd63929c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51022
I20240503 08:13:19.612679 28307 leader_election.cc:302] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a1242f3714d03a6c9ee51dd63929c, c8375219addc48fa99ac474ed1c9aef9; no voters:
I20240503 08:13:19.615252 1157 raft_consensus.cc:2781] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:19.615460 1157 raft_consensus.cc:686] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 2 LEADER]: Becoming Leader. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Running, Role: LEADER
I20240503 08:13:19.615922 1157 consensus_queue.cc:237] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } }
I20240503 08:13:19.610456 28631 raft_consensus.cc:2454] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a1242f3714d03a6c9ee51dd63929c in term 2.
I20240503 08:13:19.619012 843 raft_consensus.cc:2781] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 08:13:19.619158 843 raft_consensus.cc:482] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:19.619263 843 raft_consensus.cc:3037] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:19.626629 843 raft_consensus.cc:504] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:13:19.626889 1205 tablet_replica.cc:330] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c: stopping tablet replica
I20240503 08:13:19.627254 1205 raft_consensus.cc:2229] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:19.627554 1205 raft_consensus.cc:2258] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:19.628037 28373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba833ce23c4c4e81bc1a81125a1208c9" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "4e4a1242f3714d03a6c9ee51dd63929c"
I20240503 08:13:19.628999 28373 raft_consensus.cc:3037] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:19.629271 28502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba833ce23c4c4e81bc1a81125a1208c9" candidate_uuid: "4574396f35514c9caccb346f1a7af84d" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a"
I20240503 08:13:19.629470 28502 raft_consensus.cc:3037] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:19.632091 28373 raft_consensus.cc:2454] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 3.
I20240503 08:13:19.632608 28502 raft_consensus.cc:2454] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4574396f35514c9caccb346f1a7af84d in term 3.
I20240503 08:13:19.633917 28569 leader_election.cc:302] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4574396f35514c9caccb346f1a7af84d, 4e4a1242f3714d03a6c9ee51dd63929c, 716ffbb2a00b4f43b03e5cedcc9d302a; no voters:
I20240503 08:13:19.634338 1048 raft_consensus.cc:2781] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:19.634572 1048 raft_consensus.cc:686] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 3 LEADER]: Becoming Leader. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Running, Role: LEADER
I20240503 08:13:19.635082 1048 consensus_queue.cc:237] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:13:19.636152 843 leader_election.cc:288] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 3 election: Requested vote from peers 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529), 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397), bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937)
W20240503 08:13:19.638305 28570 leader_election.cc:334] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:13:19.648507 1205 ts_tablet_manager.cc:1875] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:19.669427 28232 catalog_manager.cc:5484] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d reported cstate change: term changed from 1 to 3, leader changed from 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) to 4574396f35514c9caccb346f1a7af84d (127.20.107.131). New cstate: current_term: 3 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:19.675760 28226 catalog_manager.cc:5484] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: term changed from 1 to 2, leader changed from 4574396f35514c9caccb346f1a7af84d (127.20.107.131) to 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129). New cstate: current_term: 2 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: NON_VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:19.683439 1205 ts_tablet_manager.cc:1888] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:13:19.685616 1205 log.cc:1196] T 9b3b8b80ebd44f27899f97211266c7b2 P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/9b3b8b80ebd44f27899f97211266c7b2
I20240503 08:13:19.686618 28218 catalog_manager.cc:4830] TS 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): tablet 9b3b8b80ebd44f27899f97211266c7b2 (table rebalance_test_table_6 [id=87e8ca2152ac403cb71e5eb67c0b2e0e]) successfully deleted
W20240503 08:13:19.719236 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:13:19.751240 30723 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 08:12:58.108057 30729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:58.108470 30729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:58.122283 30729 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:01.941489 30729 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:13:03.564656 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:03.565045 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 87e8ca2152ac403cb71e5eb67c0b2e0e: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.565387 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e4d3a510e86a4cbe8d3d611eac908187: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:03.565546 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:03.565742 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:03.565935 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2cf824bda7e41fb9ad499d46ed7c4d1: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.566066 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3a90dbe17357489db92bd286b71aa565: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:03.566195 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.566319 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:03.566454 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeb192513dc04f0eb0552ff611d314c0: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:03.566583 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.566715 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:03.566843 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e4d3a510e86a4cbe8d3d611eac908187: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:03.566968 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 87e8ca2152ac403cb71e5eb67c0b2e0e: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:03.706912 30729 rebalancer_tool.cc:1074] tablet 48e31346107a453a81921b23b244e7ff: '716ffbb2a00b4f43b03e5cedcc9d302a' -> 'bea2145b8e24436392b18ec3cfe90b93' move scheduled
I20240503 08:13:03.899960 30729 rebalancer_tool.cc:1074] tablet b2ebb0b8475b4ed9a09711a87867de08: '4574396f35514c9caccb346f1a7af84d' -> '21d21a5b596b4d48a17a1fb54a972844' move scheduled
I20240503 08:13:03.984036 30729 rebalancer_tool.cc:1084] tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3: '4e4a1242f3714d03a6c9ee51dd63929c' -> 'b29361eb4afa4ab3af7d8019d8a5042d' move ignored: Illegal state: tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3: replica b29361eb4afa4ab3af7d8019d8a5042d is already present
I20240503 08:13:03.984295 30729 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:13:06.893246 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:06.893723 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:06.893870 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3a90dbe17357489db92bd286b71aa565: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:06.894042 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:06.894152 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:06.894261 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:06.894378 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:06.894528 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:06.894616 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3a90dbe17357489db92bd286b71aa565: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:06.894830 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeb192513dc04f0eb0552ff611d314c0: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:06.894915 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:06.895035 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2cf824bda7e41fb9ad499d46ed7c4d1: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:06.895124 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e4d3a510e86a4cbe8d3d611eac908187: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:06.895242 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 87e8ca2152ac403cb71e5eb67c0b2e0e: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:06.895323 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:06.895437 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:06.895534 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:06.895627 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:07.349223 30729 rebalancer_tool.cc:1074] tablet 745d736d43244d4f9063f61de77cb473: '4e4a1242f3714d03a6c9ee51dd63929c' -> 'c8375219addc48fa99ac474ed1c9aef9' move scheduled
I20240503 08:13:07.962755 30729 rebalancer_tool.cc:1074] tablet e9997e8da99c4f45a63bc3db521b33ba: '93f7e733cec9415eb422b0a7ba8621cb' -> 'b29361eb4afa4ab3af7d8019d8a5042d' move scheduled
I20240503 08:13:08.369253 30729 rebalancer_tool.cc:1074] tablet b5736ee8065344ea8124fdf759035216: '4574396f35514c9caccb346f1a7af84d' -> 'bea2145b8e24436392b18ec3cfe90b93' move scheduled
I20240503 08:13:08.605106 30729 rebalancer_tool.cc:1074] tablet 2d113e8de02f447b9da0a1cb03686cd2: '4e4a1242f3714d03a6c9ee51dd63929c' -> '21d21a5b596b4d48a17a1fb54a972844' move scheduled
I20240503 08:13:09.014943 30729 rebalancer_tool.cc:1074] tablet d60a162155504975a39c20eff88341e1: '716ffbb2a00b4f43b03e5cedcc9d302a' -> 'b29361eb4afa4ab3af7d8019d8a5042d' move scheduled
I20240503 08:13:09.538442 30729 rebalancer_tool.cc:1074] tablet a8b4e398d34443b4a440f9ba48555d84: '4574396f35514c9caccb346f1a7af84d' -> 'c8375219addc48fa99ac474ed1c9aef9' move scheduled
I20240503 08:13:09.757383 30729 rebalancer_tool.cc:1074] tablet 28e42c3244f949c7b7343bf587894ac5: '93f7e733cec9415eb422b0a7ba8621cb' -> '21d21a5b596b4d48a17a1fb54a972844' move scheduled
I20240503 08:13:09.925117 30729 rebalancer_tool.cc:1084] tablet 09156806b0744752969b6a574e24b50a: '4e4a1242f3714d03a6c9ee51dd63929c' -> 'bea2145b8e24436392b18ec3cfe90b93' move ignored: Not found: Peer with uuid 4e4a1242f3714d03a6c9ee51dd63929c not found in consensus config
I20240503 08:13:09.925273 30729 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:13:11.788656 32535 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableSchema -> {remote=127.20.107.190:33183, user_credentials={real_user=slave}} blocked reactor thread for 52818.2us
W20240503 08:13:11.854245 32535 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.20.107.190:33183, user_credentials={real_user=slave}} blocked reactor thread for 62961.7us
W20240503 08:13:12.663074 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:12.665601 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:12.667029 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:12.667156 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeb192513dc04f0eb0552ff611d314c0: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:12.667411 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server c8375219addc48fa99ac474ed1c9aef9 to server 4574396f35514c9caccb346f1a7af84d
W20240503 08:13:12.667506 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3a90dbe17357489db92bd286b71aa565: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:12.667732 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 87e8ca2152ac403cb71e5eb67c0b2e0e: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server c8375219addc48fa99ac474ed1c9aef9
I20240503 08:13:12.928426 30729 rebalancer_tool.cc:1074] tablet b4ceeeb3495e4a8fac6e13e431c64af4: '4e4a1242f3714d03a6c9ee51dd63929c' -> 'b29361eb4afa4ab3af7d8019d8a5042d' move scheduled
I20240503 08:13:13.152838 30729 rebalancer_tool.cc:1084] tablet 4babf0ea9b954a23b1b1c4e403dce0bd: '716ffbb2a00b4f43b03e5cedcc9d302a' -> 'bea2145b8e24436392b18ec3cfe90b93' move ignored: Illegal state: tablet 4babf0ea9b954a23b1b1c4e403dce0bd: replica bea2145b8e24436392b18ec3cfe90b93 is already present
I20240503 08:13:13.153600 30729 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:13:15.034884 392 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.107.129:38529, user_credentials={real_user=slave}} blocked reactor thread for 65657.6us
W20240503 08:13:16.636384 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:16.637945 30729 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeb192513dc04f0eb0552ff611d314c0: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:16.759256 30729 rebalancer_tool.cc:1084] tablet d44335d8be7245b1bb52b2eab9e42c18: '716ffbb2a00b4f43b03e5cedcc9d302a' -> '' move ignored: Not found: Peer with uuid 716ffbb2a00b4f43b03e5cedcc9d302a not found in consensus config
I20240503 08:13:16.759426 30729 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:13:18.438506 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:13:17.928718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.107.135:45937 (local address 127.0.0.1:34266)
0503 08:13:17.928996 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:17.929014 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:17.929110 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:18.437003 (+507893us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":145}
Illegal state: tablet server bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): unacceptable health status UNAVAILABLE
I20240503 08:13:19.863102 1048 raft_consensus.cc:1050] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d to VOTER
I20240503 08:13:19.864243 1048 consensus_queue.cc:237] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:19.871928 30727 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 08:12:57.781622 30733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:57.782106 30733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:57.795500 30733 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:01.151221 30733 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:13:03.322374 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3a90dbe17357489db92bd286b71aa565: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.322690 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:03.323603 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:03.323768 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 87e8ca2152ac403cb71e5eb67c0b2e0e: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.325418 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:03.325578 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e4d3a510e86a4cbe8d3d611eac908187: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:03.325683 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2cf824bda7e41fb9ad499d46ed7c4d1: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:03.325798 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:03.356508 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeb192513dc04f0eb0552ff611d314c0: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.356729 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.356849 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:03.356997 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 87e8ca2152ac403cb71e5eb67c0b2e0e: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:03.357089 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.357185 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server c8375219addc48fa99ac474ed1c9aef9
I20240503 08:13:03.633159 30733 rebalancer_tool.cc:1074] tablet b2ebb0b8475b4ed9a09711a87867de08: '716ffbb2a00b4f43b03e5cedcc9d302a' -> 'bea2145b8e24436392b18ec3cfe90b93' move scheduled
I20240503 08:13:03.946549 30733 rebalancer_tool.cc:1074] tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3: '4e4a1242f3714d03a6c9ee51dd63929c' -> '21d21a5b596b4d48a17a1fb54a972844' move scheduled
I20240503 08:13:04.121963 30733 rebalancer_tool.cc:1074] tablet 320d1e880958427a88e93321171cb36e: '4574396f35514c9caccb346f1a7af84d' -> 'b29361eb4afa4ab3af7d8019d8a5042d' move scheduled
I20240503 08:13:04.502339 30733 rebalancer_tool.cc:1074] tablet 585ba864cc384e7f9db90e9d4c47e82c: '93f7e733cec9415eb422b0a7ba8621cb' -> 'c8375219addc48fa99ac474ed1c9aef9' move scheduled
I20240503 08:13:04.833196 30733 rebalancer_tool.cc:1074] tablet ede24a27608a4cf196dd46f85c4163e1: '716ffbb2a00b4f43b03e5cedcc9d302a' -> 'bea2145b8e24436392b18ec3cfe90b93' move scheduled
I20240503 08:13:05.259938 30733 rebalancer_tool.cc:1074] tablet ac392ff7965a405f96efc777aca4f0bb: '4e4a1242f3714d03a6c9ee51dd63929c' -> 'b29361eb4afa4ab3af7d8019d8a5042d' move scheduled
I20240503 08:13:05.493741 30733 rebalancer_tool.cc:1074] tablet 4babf0ea9b954a23b1b1c4e403dce0bd: '4574396f35514c9caccb346f1a7af84d' -> '21d21a5b596b4d48a17a1fb54a972844' move scheduled
I20240503 08:13:05.769591 30733 rebalancer_tool.cc:1074] tablet b0d07a6278d545e69bf7d2012109d1f4: '93f7e733cec9415eb422b0a7ba8621cb' -> 'c8375219addc48fa99ac474ed1c9aef9' move scheduled
I20240503 08:13:06.170792 30733 rebalancer_tool.cc:1074] tablet b4ceeeb3495e4a8fac6e13e431c64af4: '716ffbb2a00b4f43b03e5cedcc9d302a' -> 'bea2145b8e24436392b18ec3cfe90b93' move scheduled
I20240503 08:13:06.514215 30733 rebalancer_tool.cc:1074] tablet 09156806b0744752969b6a574e24b50a: '4e4a1242f3714d03a6c9ee51dd63929c' -> 'b29361eb4afa4ab3af7d8019d8a5042d' move scheduled
I20240503 08:13:06.774809 30733 rebalancer_tool.cc:1084] tablet 0caf86850b3841ea811b0a6ab19ae330: '4574396f35514c9caccb346f1a7af84d' -> '21d21a5b596b4d48a17a1fb54a972844' move ignored: Illegal state: tablet 0caf86850b3841ea811b0a6ab19ae330: replica 21d21a5b596b4d48a17a1fb54a972844 is already present
I20240503 08:13:06.775010 30733 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:13:10.224257 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:10.224635 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3a90dbe17357489db92bd286b71aa565: could not find any suitable replica to move from server b29361eb4afa4ab3af7d8019d8a5042d to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:10.243296 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:10.248314 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:10.251592 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:10.255594 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:10.257362 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:10.261441 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:10.262003 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:10.269963 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeb192513dc04f0eb0552ff611d314c0: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:10.274374 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:10.275408 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeb192513dc04f0eb0552ff611d314c0: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:10.275969 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:10.596298 30733 rebalancer_tool.cc:1084] tablet a8b4e398d34443b4a440f9ba48555d84: '4574396f35514c9caccb346f1a7af84d' -> 'c8375219addc48fa99ac474ed1c9aef9' move ignored: Illegal state: tablet a8b4e398d34443b4a440f9ba48555d84: replica c8375219addc48fa99ac474ed1c9aef9 is already present
I20240503 08:13:10.596436 30733 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:13:14.745965 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 87e8ca2152ac403cb71e5eb67c0b2e0e: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:14.746424 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:14.746590 30733 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:15.890976 30733 rebalancer_tool.cc:1074] tablet d44335d8be7245b1bb52b2eab9e42c18: '716ffbb2a00b4f43b03e5cedcc9d302a' -> '' move scheduled
I20240503 08:13:16.633700 30733 rebalancer_tool.cc:1074] tablet c6e1b9f104ea471cab7527bd56433327: '93f7e733cec9415eb422b0a7ba8621cb' -> '4574396f35514c9caccb346f1a7af84d' move scheduled
I20240503 08:13:16.766042 30733 rebalancer_tool.cc:1074] tablet d13c33b8712e4c488abb99b73ff42b86: '4e4a1242f3714d03a6c9ee51dd63929c' -> '21d21a5b596b4d48a17a1fb54a972844' move scheduled
I20240503 08:13:16.917168 30733 rebalancer_tool.cc:1074] tablet 173376257c4946b5a801a27e4dffe90d: '4574396f35514c9caccb346f1a7af84d' -> 'bea2145b8e24436392b18ec3cfe90b93' move scheduled
I20240503 08:13:17.251124 30733 rebalancer_tool.cc:1074] tablet 2d113e8de02f447b9da0a1cb03686cd2: '93f7e733cec9415eb422b0a7ba8621cb' -> 'b29361eb4afa4ab3af7d8019d8a5042d' move scheduled
I20240503 08:13:17.550935 30733 rebalancer_tool.cc:1084] tablet b5736ee8065344ea8124fdf759035216: '93f7e733cec9415eb422b0a7ba8621cb' -> '21d21a5b596b4d48a17a1fb54a972844' move ignored: Illegal state: tablet b5736ee8065344ea8124fdf759035216: replica 21d21a5b596b4d48a17a1fb54a972844 is already present
I20240503 08:13:17.551097 30733 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): unacceptable health status UNAVAILABLE
W20240503 08:13:19.874198 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:13:19.876950 30676 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:19.877621 1204 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:19.881176 30414 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:19.881821 1204 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:19.892967 28762 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:13:19.893625 843 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:13:19.937211 30725 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 08:12:57.890503 30731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:57.890954 30731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:57.903656 30731 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:01.011687 30731 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:13:03.413875 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeb192513dc04f0eb0552ff611d314c0: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:03.414261 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:03.423619 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:03.423869 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:03.423969 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.424060 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 87e8ca2152ac403cb71e5eb67c0b2e0e: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.424135 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.424237 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e4d3a510e86a4cbe8d3d611eac908187: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:03.424314 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2cf824bda7e41fb9ad499d46ed7c4d1: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:03.424423 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3a90dbe17357489db92bd286b71aa565: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:03.424546 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:03.424726 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.424822 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:03.424952 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 87e8ca2152ac403cb71e5eb67c0b2e0e: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:03.655164 30731 rebalancer_tool.cc:1074] tablet c6e1b9f104ea471cab7527bd56433327: '4574396f35514c9caccb346f1a7af84d' -> 'c8375219addc48fa99ac474ed1c9aef9' move scheduled
I20240503 08:13:03.854630 30731 rebalancer_tool.cc:1084] tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3: '4e4a1242f3714d03a6c9ee51dd63929c' -> '21d21a5b596b4d48a17a1fb54a972844' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:13:03.854811 30731 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:13:06.853096 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:06.853562 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:06.857185 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3a90dbe17357489db92bd286b71aa565: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:06.857753 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeb192513dc04f0eb0552ff611d314c0: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:06.857892 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e4d3a510e86a4cbe8d3d611eac908187: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:06.857986 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:06.858083 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:06.858166 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3a90dbe17357489db92bd286b71aa565: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:06.858870 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:06.858991 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:06.859140 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:06.859381 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:06.859516 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2cf824bda7e41fb9ad499d46ed7c4d1: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:06.859606 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 87e8ca2152ac403cb71e5eb67c0b2e0e: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:06.859705 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e4d3a510e86a4cbe8d3d611eac908187: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:06.859791 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:06.859892 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:06.859974 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server c8375219addc48fa99ac474ed1c9aef9
I20240503 08:13:07.137238 30731 rebalancer_tool.cc:1084] tablet 09156806b0744752969b6a574e24b50a: '4e4a1242f3714d03a6c9ee51dd63929c' -> 'b29361eb4afa4ab3af7d8019d8a5042d' move ignored: Illegal state: tablet 09156806b0744752969b6a574e24b50a: replica b29361eb4afa4ab3af7d8019d8a5042d is already present
I20240503 08:13:07.137369 30731 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:13:10.529119 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:10.529318 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:10.529444 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:10.529531 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:10.529692 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3a90dbe17357489db92bd286b71aa565: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:10.529837 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeb192513dc04f0eb0552ff611d314c0: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:10.529973 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:10.530071 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeb192513dc04f0eb0552ff611d314c0: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:10.530223 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e4d3a510e86a4cbe8d3d611eac908187: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server c8375219addc48fa99ac474ed1c9aef9
I20240503 08:13:10.910813 30731 rebalancer_tool.cc:1074] tablet ba833ce23c4c4e81bc1a81125a1208c9: '4e4a1242f3714d03a6c9ee51dd63929c' -> 'bea2145b8e24436392b18ec3cfe90b93' move scheduled
I20240503 08:13:11.736265 30731 rebalancer_tool.cc:1074] tablet 9b3b8b80ebd44f27899f97211266c7b2: '716ffbb2a00b4f43b03e5cedcc9d302a' -> '21d21a5b596b4d48a17a1fb54a972844' move scheduled
I20240503 08:13:12.193833 30731 rebalancer_tool.cc:1074] tablet da529c8f8de241d38678d1596b9402ac: '93f7e733cec9415eb422b0a7ba8621cb' -> 'b29361eb4afa4ab3af7d8019d8a5042d' move scheduled
I20240503 08:13:12.424935 30731 rebalancer_tool.cc:1074] tablet 564fb1b51abd4f8da0372ec99e3c599c: '4574396f35514c9caccb346f1a7af84d' -> 'c8375219addc48fa99ac474ed1c9aef9' move scheduled
I20240503 08:13:12.654259 30731 rebalancer_tool.cc:1074] tablet 173376257c4946b5a801a27e4dffe90d: '4e4a1242f3714d03a6c9ee51dd63929c' -> '21d21a5b596b4d48a17a1fb54a972844' move scheduled
I20240503 08:13:12.889428 30731 rebalancer_tool.cc:1074] tablet 4babf0ea9b954a23b1b1c4e403dce0bd: '716ffbb2a00b4f43b03e5cedcc9d302a' -> 'bea2145b8e24436392b18ec3cfe90b93' move scheduled
I20240503 08:13:13.206274 30731 rebalancer_tool.cc:1074] tablet 0caf86850b3841ea811b0a6ab19ae330: '93f7e733cec9415eb422b0a7ba8621cb' -> '4574396f35514c9caccb346f1a7af84d' move scheduled
I20240503 08:13:13.385103 30731 rebalancer_tool.cc:1074] tablet a20489bf65774902af38d228ee966ffe: '4e4a1242f3714d03a6c9ee51dd63929c' -> 'b29361eb4afa4ab3af7d8019d8a5042d' move scheduled
I20240503 08:13:13.634006 30731 rebalancer_tool.cc:1074] tablet ac392ff7965a405f96efc777aca4f0bb: '93f7e733cec9415eb422b0a7ba8621cb' -> 'c8375219addc48fa99ac474ed1c9aef9' move scheduled
I20240503 08:13:13.882851 30731 rebalancer_tool.cc:1084] tablet 28e42c3244f949c7b7343bf587894ac5: '4e4a1242f3714d03a6c9ee51dd63929c' -> '21d21a5b596b4d48a17a1fb54a972844' move ignored: Illegal state: tablet 28e42c3244f949c7b7343bf587894ac5: replica 21d21a5b596b4d48a17a1fb54a972844 is already present
I20240503 08:13:13.882972 30731 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:13:16.820215 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server b29361eb4afa4ab3af7d8019d8a5042d to server 93f7e733cec9415eb422b0a7ba8621cb
W20240503 08:13:16.820410 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3a90dbe17357489db92bd286b71aa565: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:16.820524 30731 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:17.023712 30731 rebalancer_tool.cc:1084] tablet b5736ee8065344ea8124fdf759035216: '93f7e733cec9415eb422b0a7ba8621cb' -> '21d21a5b596b4d48a17a1fb54a972844' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:13:17.023867 30731 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:13:18.440038 992 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:13:17.974809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.107.135:45937 (local address 127.0.0.1:34268)
0503 08:13:18.048454 (+ 73645us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:18.048475 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:18.048584 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:18.438893 (+390309us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":67207}
Illegal state: tablet server bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): unacceptable health status UNAVAILABLE
I20240503 08:13:19.940302 1204 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:19.945341 30412 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:19.953473 30672 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:19.964013 28226 catalog_manager.cc:5484] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: config changed from index 10 to 11, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:19.967384 28762 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:19.977347 28631 consensus_queue.cc:237] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:13:19.988137 30412 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:13:19.989003 1204 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20240503 08:13:19.996729 1048 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:20.003033 30414 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:20.005450 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d113e8de02f447b9da0a1cb03686cd2 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:20.008960 28232 catalog_manager.cc:5484] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 11 to 12, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:20.012295 30676 raft_consensus.cc:1261] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:13:20.013239 1204 consensus_queue.cc:1035] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:13:20.016660 28744 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d113e8de02f447b9da0a1cb03686cd2 with delete_type TABLET_DATA_TOMBSTONED (TS 93f7e733cec9415eb422b0a7ba8621cb not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:39918
I20240503 08:13:20.015496 30672 raft_consensus.cc:2932] T 2d113e8de02f447b9da0a1cb03686cd2 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:13:20.021405 1235 tablet_replica.cc:330] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb: stopping tablet replica
I20240503 08:13:20.021683 1235 raft_consensus.cc:2229] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:20.022296 1235 raft_consensus.cc:2258] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:20.028213 1235 ts_tablet_manager.cc:1875] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:20.034806 28502 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 3 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 08:13:20.035732 1204 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:20.042016 1235 ts_tablet_manager.cc:1888] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240503 08:13:20.042225 1235 log.cc:1196] T 2d113e8de02f447b9da0a1cb03686cd2 P 93f7e733cec9415eb422b0a7ba8621cb: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-3/wal/wals/2d113e8de02f447b9da0a1cb03686cd2
I20240503 08:13:20.043196 28218 catalog_manager.cc:4830] TS 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095): tablet 2d113e8de02f447b9da0a1cb03686cd2 (table rebalance_test_table_8 [id=649380be96b24c78aa15b40471703d93]) successfully deleted
I20240503 08:13:20.094319 30726 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskZB6CEZ/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 08:12:58.028553 30732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:58.035880 30732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:58.066329 30732 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:01.845752 30732 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:13:03.614408 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e4d3a510e86a4cbe8d3d611eac908187: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.614753 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:03.614876 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:03.615010 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:03.615131 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 87e8ca2152ac403cb71e5eb67c0b2e0e: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:03.615252 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:03.615491 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2cf824bda7e41fb9ad499d46ed7c4d1: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:03.615613 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.615713 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3a90dbe17357489db92bd286b71aa565: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.615839 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeb192513dc04f0eb0552ff611d314c0: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:03.615990 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:03.616101 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e4d3a510e86a4cbe8d3d611eac908187: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:03.616219 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:03.616343 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 87e8ca2152ac403cb71e5eb67c0b2e0e: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:03.831748 30732 rebalancer_tool.cc:1074] tablet b2ebb0b8475b4ed9a09711a87867de08: '716ffbb2a00b4f43b03e5cedcc9d302a' -> 'c8375219addc48fa99ac474ed1c9aef9' move scheduled
I20240503 08:13:03.957197 30732 rebalancer_tool.cc:1084] tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3: '4e4a1242f3714d03a6c9ee51dd63929c' -> 'b29361eb4afa4ab3af7d8019d8a5042d' move ignored: Illegal state: tablet 3bfa8537bc8a4950b53fbf3c36d5ecf3: replica b29361eb4afa4ab3af7d8019d8a5042d is already present
I20240503 08:13:03.957356 30732 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:13:07.088842 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 87e8ca2152ac403cb71e5eb67c0b2e0e: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:07.093438 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:07.095289 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3a90dbe17357489db92bd286b71aa565: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:07.095467 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:07.095577 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeb192513dc04f0eb0552ff611d314c0: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:07.095696 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:07.095847 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:07.095963 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:07.096138 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:07.096328 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:07.096455 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2cf824bda7e41fb9ad499d46ed7c4d1: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:07.096577 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 87e8ca2152ac403cb71e5eb67c0b2e0e: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:07.096705 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e4d3a510e86a4cbe8d3d611eac908187: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:07.096827 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ff6aeab7e1847f6884750db73f8c428: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:07.096998 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:07.098078 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3a90dbe17357489db92bd286b71aa565: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:13:08.169016 30732 rebalancer_tool.cc:1074] tablet c6e1b9f104ea471cab7527bd56433327: '4574396f35514c9caccb346f1a7af84d' -> 'b29361eb4afa4ab3af7d8019d8a5042d' move scheduled
I20240503 08:13:08.487797 30732 rebalancer_tool.cc:1074] tablet 2d113e8de02f447b9da0a1cb03686cd2: '4e4a1242f3714d03a6c9ee51dd63929c' -> 'bea2145b8e24436392b18ec3cfe90b93' move scheduled
I20240503 08:13:08.882647 30732 rebalancer_tool.cc:1074] tablet e9997e8da99c4f45a63bc3db521b33ba: '93f7e733cec9415eb422b0a7ba8621cb' -> '21d21a5b596b4d48a17a1fb54a972844' move scheduled
I20240503 08:13:09.216085 30732 rebalancer_tool.cc:1074] tablet a8a9adc3712d436c95505a3216e7607c: '716ffbb2a00b4f43b03e5cedcc9d302a' -> 'c8375219addc48fa99ac474ed1c9aef9' move scheduled
I20240503 08:13:09.615650 30732 rebalancer_tool.cc:1074] tablet b5736ee8065344ea8124fdf759035216: '4574396f35514c9caccb346f1a7af84d' -> 'b29361eb4afa4ab3af7d8019d8a5042d' move scheduled
I20240503 08:13:09.909701 30732 rebalancer_tool.cc:1084] tablet d13c33b8712e4c488abb99b73ff42b86: '4e4a1242f3714d03a6c9ee51dd63929c' -> 'bea2145b8e24436392b18ec3cfe90b93' move ignored: Illegal state: tablet d13c33b8712e4c488abb99b73ff42b86: replica bea2145b8e24436392b18ec3cfe90b93 is already present
I20240503 08:13:09.909869 30732 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:13:11.259518 32489 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.20.107.190:33183, user_credentials={real_user=slave}} blocked reactor thread for 116978us
W20240503 08:13:12.344597 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeb192513dc04f0eb0552ff611d314c0: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:12.357504 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 649380be96b24c78aa15b40471703d93: could not find any suitable replica to move from server 93f7e733cec9415eb422b0a7ba8621cb to server b29361eb4afa4ab3af7d8019d8a5042d
W20240503 08:13:12.359436 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b9e797909b184d0bb18e1af0519e9966: could not find any suitable replica to move from server 21d21a5b596b4d48a17a1fb54a972844 to server 4574396f35514c9caccb346f1a7af84d
W20240503 08:13:12.359578 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3a90dbe17357489db92bd286b71aa565: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server bea2145b8e24436392b18ec3cfe90b93
W20240503 08:13:12.360424 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0225804a5c4a42cbb32c4b0029752bd1: could not find any suitable replica to move from server 4574396f35514c9caccb346f1a7af84d to server c8375219addc48fa99ac474ed1c9aef9
W20240503 08:13:12.362635 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeb192513dc04f0eb0552ff611d314c0: could not find any suitable replica to move from server 4e4a1242f3714d03a6c9ee51dd63929c to server 21d21a5b596b4d48a17a1fb54a972844
W20240503 08:13:12.363139 30732 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 311fe8a0590b44a8a07f0203d8f65e32: could not find any suitable replica to move from server 716ffbb2a00b4f43b03e5cedcc9d302a to server bea2145b8e24436392b18ec3cfe90b93
I20240503 08:13:12.611442 30732 rebalancer_tool.cc:1074] tablet 585ba864cc384e7f9db90e9d4c47e82c: '4e4a1242f3714d03a6c9ee51dd63929c' -> '21d21a5b596b4d48a17a1fb54a972844' move scheduled
I20240503 08:13:12.767838 30732 rebalancer_tool.cc:1074] tablet 0caf86850b3841ea811b0a6ab19ae330: '93f7e733cec9415eb422b0a7ba8621cb' -> 'bea2145b8e24436392b18ec3cfe90b93' move scheduled
I20240503 08:13:13.086686 30732 rebalancer_tool.cc:1074] tablet a20489bf65774902af38d228ee966ffe: '716ffbb2a00b4f43b03e5cedcc9d302a' -> 'c8375219addc48fa99ac474ed1c9aef9' move scheduled
I20240503 08:13:13.328464 30732 rebalancer_tool.cc:1084] tablet 173376257c4946b5a801a27e4dffe90d: '4574396f35514c9caccb346f1a7af84d' -> '21d21a5b596b4d48a17a1fb54a972844' move ignored: Illegal state: tablet 173376257c4946b5a801a27e4dffe90d: replica 21d21a5b596b4d48a17a1fb54a972844 is already present
I20240503 08:13:13.328625 30732 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:13:15.473843 548 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.20.107.190:33183, user_credentials={real_user=slave}} blocked reactor thread for 56854.3us
I20240503 08:13:17.134323 30732 rebalancer_tool.cc:1084] tablet 173376257c4946b5a801a27e4dffe90d: '4574396f35514c9caccb346f1a7af84d' -> '' move ignored: Not found: Peer with uuid 4574396f35514c9caccb346f1a7af84d not found in consensus config
I20240503 08:13:17.134480 30732 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): unacceptable health status UNAVAILABLE
I20240503 08:13:20.121204 28631 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:20.122185 955 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:13:20.123013 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:20.153400 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:20.155292 30280 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P c8375219addc48fa99ac474ed1c9aef9 [term 2 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:20.156670 1157 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:13:20.191972 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:20.261878 28373 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 3 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 08:13:20.262563 1204 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:13:20.350317 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:20.478185 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:20.537840 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:20.629873 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:20.844846 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:20.861960 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:20.881821 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:20.903215 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:20.950284 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:20.965397 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:21.323091 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:21.465493 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:21.474620 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:21.494725 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:21.828164 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:22.021724 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:22.360226 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:22.575640 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:22.609661 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:22.773386 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:22.802659 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:22.868358 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:22.913126 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:22.962770 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:23.298091 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:23.373558 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:23.400386 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:23.454767 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:13:23.454821 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 08:13:23.492204 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:23.573918 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:23.691006 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240503 08:13:23.728943 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:23.823977 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:23.929291 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:23.966786 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:23.980860 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:13:24.167907 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:24.446770 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:24.549259 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:24.933055 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:24.945645 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:25.225564 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:25.242236 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:13:25.348806 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:25.394485 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:25.442318 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:25.486657 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:13:25.843922 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:25.932475 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:25.934000 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:13:26.024842 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:13:26.142272 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:13:26.227728 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:13:26.277235 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:26.285622 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:13:26.289186 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:13:26.459542 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:26.811229 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:26.934278 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:27.339555 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:27.344254 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:27.787020 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:13:27.802793 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:27.806507 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:13:27.814841 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:27.962041 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:13:28.069804 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:13:28.265136 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:28.282255 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:28.444096 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:13:28.466356 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:13:28.501008 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:13:28.501775 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:13:28.711815 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:13:28.777230 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:13:28.814306 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:28.857226 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:28.913964 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:13:28.952503 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:13:28.962239 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:13:29.165791 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:13:29.274768 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:29.457813 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:29.704504 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:30.005515 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:30.202842 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:30.228185 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:13:30.384925 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:13:30.427076 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:13:30.570734 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:30.635181 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:13:30.724505 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:30.902695 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:13:30.971738 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:13:31.055291 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:13:31.113970 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:31.159049 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:13:31.286504 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:31.366206 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:13:31.536046 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:13:31.650709 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:31.708635 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:32.076124 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:32.250566 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:32.468017 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:13:32.536275 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:32.637921 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:32.788367 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:13:32.858520 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:13:33.130877 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:33.135699 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:33.321900 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:13:33.325515 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:13:33.522940 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:13:33.586386 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:13:33.616225 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:33.687412 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:33.713797 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:13:33.714742 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:13:33.744436 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:13:33.847052 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:13:33.954236 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 08:13:34.148897 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:34.184137 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:34.359725 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:13:34.633985 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:34.656123 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:34.778316 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:13:35.089545 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:35.143653 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:13:35.225772 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:35.385798 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:13:35.482564 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:13:35.533515 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:35.737823 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:35.880205 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:13:35.967008 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:36.082932 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:13:36.109393 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:13:36.137748 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:13:36.221503 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:36.353693 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:13:36.427599 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:36.536181 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:13:36.687121 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:36.896822 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:13:36.953044 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:37.075054 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:37.526137 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:37.564039 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:37.917498 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:13:37.930755 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:13:37.931315 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:38.052997 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:38.076221 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:13:38.347155 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:13:38.437454 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:13:38.490139 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:38.531131 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:38.590607 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:13:38.811650 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:13:38.825863 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:13:38.932790 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:13:39.023380 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:39.039297 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:13:39.061148 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:13:39.077538 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:39.280043 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:13:39.280980 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:13:39.405840 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:39.602672 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:39.875907 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:40.096532 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:40.101975 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:13:40.419962 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:13:40.484143 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:40.487223 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:40.542685 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:13:40.622298 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:13:40.929842 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:13:40.955538 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:41.017791 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:13:41.045058 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:13:41.072345 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:41.261866 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:13:41.477375 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:13:41.548000 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:41.567806 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:41.661667 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:13:41.829862 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:13:42.105535 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:42.157729 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:42.608243 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:42.691979 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:43.031719 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:43.111140 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:13:43.142925 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:13:43.224292 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:13:43.257331 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:43.375460 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:13:43.546275 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:13:43.579190 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:43.602463 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:13:43.638978 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:43.741792 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:13:43.942173 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:13:43.942963 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:13:44.035696 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:13:44.098269 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:13:44.178332 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:44.193897 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:44.363127 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:13:44.399380 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:13:44.715142 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:44.743150 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:45.179327 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:13:45.219451 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:45.243858 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:45.564390 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:13:45.669335 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:45.669466 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:45.728030 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:13:45.747853 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:13:45.772179 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:13:45.906064 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:13:46.084686 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:13:46.251466 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:46.282829 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:46.390763 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:13:46.494969 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:13:46.662859 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:13:46.747340 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:46.785185 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:47.180681 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:13:47.260799 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:47.342643 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:47.820573 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:47.856563 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:13:47.895097 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:13:47.902002 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:48.138267 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:13:48.235988 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:48.281955 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:13:48.376540 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:13:48.493860 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:48.545661 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:13:48.663287 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:48.992578 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:48.998759 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:13:49.028569 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:13:49.101115 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:13:49.237237 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:13:49.242386 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:13:49.243350 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:49.372787 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:49.589413 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:13:49.606882 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:13:49.758821 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:49.771775 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:50.359483 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:50.364467 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:50.464351 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:13:50.477244 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:13:50.478535 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:13:50.495584 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:13:50.551698 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:13:50.713944 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:13:50.810149 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:50.987381 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:51.245700 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:13:51.378551 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:51.408708 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:51.451861 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:13:51.509148 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:13:51.738859 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:13:51.798705 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:51.962026 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:52.018435 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:13:52.310549 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:52.456593 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:52.739241 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:52.790657 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:13:52.953723 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:13:52.969552 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:52.970657 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:13:52.977794 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:13:53.243440 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:53.246721 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:13:53.507818 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:53.814256 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:13:53.838845 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:54.019150 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:13:54.048900 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:13:54.049945 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:13:54.089809 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:54.107230 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:13:54.324339 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:13:54.325388 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:54.490469 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:54.633656 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:13:54.717443 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:13:54.869169 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:54.905086 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:55.255810 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:13:55.315663 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:55.523160 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:13:55.528393 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:13:55.528965 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:55.529603 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:13:55.560318 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:13:55.578891 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:13:55.855734 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:56.130335 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:56.268539 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:13:56.291824 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:13:56.332446 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:56.597476 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:13:56.661926 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:56.706720 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:13:56.936609 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:57.129247 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:57.255160 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:13:57.383797 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:57.556735 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:57.637761 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:13:57.870466 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:13:57.883018 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:58.012797 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:13:58.181059 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:58.230648 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:13:58.240233 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:13:58.334247 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:58.785792 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:58.823141 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:13:58.872637 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:13:58.900678 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:59.184185 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240503 08:13:59.202400 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:13:59.220090 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:13:59.286769 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:59.355237 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:13:59.431800 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:13:59.639521 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:13:59.731406 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:13:59.830768 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:00.029587 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:00.168720 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:14:00.211861 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:14:00.276450 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:00.363822 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:14:00.637856 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:00.695844 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:14:00.719264 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:00.790731 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:14:01.012421 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:14:01.205901 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:01.226126 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:14:01.318178 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:14:01.322646 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:01.346573 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:14:01.639592 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:01.761684 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:01.927268 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:14:02.151578 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:02.195349 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:14:02.295435 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:02.673770 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:14:02.705385 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:02.706128 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:14:02.747689 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:14:02.766911 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:03.212283 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:03.369128 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:03.453603 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:14:03.561173 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:14:03.593307 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:03.623107 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:14:03.697906 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:14:03.811882 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:03.837157 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:14:04.189873 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:04.303995 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:04.447824 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:14:04.458274 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:14:04.459105 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:14:04.609571 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:14:04.690102 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:04.709043 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:04.763159 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:14:05.071257 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:05.083146 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:14:05.153365 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:05.234722 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:14:05.266789 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:14:05.447940 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:05.708014 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:05.881414 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:14:05.933399 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:14:06.021046 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:06.065992 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:14:06.107015 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:14:06.126149 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:14:06.174953 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:14:06.222810 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:06.529086 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:06.609623 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:06.678259 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:14:07.090126 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:14:07.118111 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:07.152539 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:07.579066 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:14:07.638790 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:07.710641 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:07.845492 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:14:08.023936 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:08.147293 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:14:08.221818 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:08.294154 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:14:08.354846 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:14:08.462069 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:14:08.505502 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:08.703969 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:14:08.781919 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:08.839458 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:14:08.900825 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:09.126009 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:14:09.332662 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:09.463358 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:14:09.464257 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:09.680639 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:14:09.918620 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:09.919550 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:14:09.946909 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:14:10.004385 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:10.156369 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:14:10.339426 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:14:10.382494 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:10.460371 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:10.667065 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:14:10.765938 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:14:10.794706 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:11.013034 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:14:11.046887 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:14:11.049803 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:14:11.063777 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:11.128851 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:14:11.179157 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:11.230381 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:14:11.514391 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:11.573096 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:11.662570 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:14:11.898717 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:12.012626 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:12.351485 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:12.455336 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:12.515498 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:14:12.686134 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:14:12.708734 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:14:12.904081 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:14:12.918803 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:12.937358 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:13.388214 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:14:13.472038 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:14:13.486562 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:13.504249 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:14:13.522970 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:13.800240 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:14:13.826177 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:14:13.864216 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:13.913949 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:14.255996 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:14:14.310391 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:14.385568 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:14.635500 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 08:14:14.694403 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:14:14.775955 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:14:14.813517 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:14.854702 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:14.980626 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 08:14:15.108072 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:14:15.222756 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:14:15.256269 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:15.311208 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:14:15.369246 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:15.665339 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:15.832434 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:14:15.855010 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:16.054744 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:14:16.074435 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:14:16.158252 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:16.221273 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:14:16.272668 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:16.345297 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:14:16.470825 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:14:16.655179 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:14:16.699312 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:16.723790 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:17.039584 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:14:17.087285 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:17.122370 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:17.445102 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:14:17.513919 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:17.550504 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:17.690526 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:14:17.762099 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:14:17.942983 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:17.966478 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:18.265199 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:14:18.405508 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:18.462713 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:14:18.565407 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:14:18.571045 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:18.590279 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:14:18.746994 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:14:19.009372 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:19.087756 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:19.157125 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:14:19.527370 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:14:19.560082 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:19.592494 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:19.771930 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:14:19.792099 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 08:14:19.792917 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:14:20.042279 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:14:20.062763 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:20.084537 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:20.378170 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:14:20.438553 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:14:20.475468 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:20.526216 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:20.858812 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:14:20.866801 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:20.931358 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:20.937666 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:14:20.998445 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:14:21.046159 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:14:21.114502 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:14:21.308321 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:21.347520 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:14:21.489328 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:21.624215 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:14:21.754930 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:21.902109 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:21.994228 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:14:22.215174 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:22.457289 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:14:22.482686 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:22.739318 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:22.871466 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:14:22.911409 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:14:23.049779 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:23.294863 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:23.351886 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:14:23.383047 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:14:23.461503 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:14:23.547189 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:23.547477 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:14:23.721388 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:14:23.724507 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:23.858260 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:14:23.950192 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:24.102972 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:24.398277 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:24.667124 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:14:24.694217 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:24.833026 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:14:24.910465 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:14:25.007193 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:25.119091 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:25.218883 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:14:25.286263 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:14:25.286955 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:14:25.413015 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:25.641914 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:14:25.645694 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:25.820519 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:14:25.904408 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:14:25.956311 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:25.983326 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:14:26.127094 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:14:26.134321 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:14:26.192826 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:26.259042 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:14:26.512632 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:26.676151 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:14:26.677268 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:26.971558 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:27.054133 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:14:27.269870 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:27.448835 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:27.691010 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:14:27.742893 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:27.748960 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:14:27.950616 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:14:28.004998 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:28.119040 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:28.295140 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:14:28.377323 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:14:28.417832 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:14:28.435739 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:28.445663 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:14:28.542026 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:14:28.585765 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:28.844166 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:28.902151 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:14:28.987211 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:29.222892 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:29.407820 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:29.750169 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:14:29.782073 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:29.962651 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:14:29.988519 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:14:29.989356 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:29.999079 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:14:30.263301 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:14:30.267431 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:14:30.309763 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:14:30.356122 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:30.441097 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:30.722018 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:14:30.763342 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:14:30.790732 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:14:30.853464 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:30.908545 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:30.977979 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:14:31.109970 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:14:31.363266 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:31.447854 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:31.677614 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:14:31.702548 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:14:31.754155 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:32.051656 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:32.139904 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:14:32.263257 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:14:32.347115 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:32.657178 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:32.773588 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:14:32.791680 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:14:32.936250 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:33.227969 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:33.231447 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:14:33.295980 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:14:33.303820 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:14:33.477006 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:33.500697 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:14:33.515620 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:14:33.798501 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:34.101943 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:34.227545 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:34.302682 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:14:34.365321 28222 maintenance_manager.cc:419] P e8f5acc8744f4dccbfd654a6d7dab5bf: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033370
I20240503 08:14:34.409655 28221 maintenance_manager.cc:643] P e8f5acc8744f4dccbfd654a6d7dab5bf: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.044s user 0.032s sys 0.008s Metrics: {"bytes_written":20802,"cfile_init":1,"dirs.queue_time_us":337,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1125,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":952,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"rows_written":43,"thread_start_us":6728,"threads_started":2,"wal-append.queue_time_us":6336}
I20240503 08:14:34.410910 28222 maintenance_manager.cc:419] P e8f5acc8744f4dccbfd654a6d7dab5bf: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 10322 bytes on disk
I20240503 08:14:34.412158 28221 maintenance_manager.cc:643] P e8f5acc8744f4dccbfd654a6d7dab5bf: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240503 08:14:34.685112 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:34.756366 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:14:34.825740 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:34.881032 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:14:35.012521 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:14:35.143285 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:14:35.168725 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:14:35.227802 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:14:35.234751 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:35.386061 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:35.695534 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:14:35.696381 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:14:35.697552 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:35.813076 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:14:35.870723 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:14:35.993731 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:14:36.001420 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:36.199479 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:36.276360 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:14:36.382107 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:36.585435 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:36.746685 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:14:36.781198 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:36.988916 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:14:37.187300 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:37.310508 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:14:37.327692 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:37.444600 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:14:37.633095 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:37.767764 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:14:37.836266 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:38.037633 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:14:38.068043 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:38.143047 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:14:38.266283 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:14:38.291514 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:14:38.358618 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:38.499910 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:38.583164 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:14:38.849407 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:38.863454 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:14:38.985587 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:39.239630 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:39.503013 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:39.692953 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:39.723766 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:14:39.743857 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:14:40.026943 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:14:40.031486 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:14:40.118839 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:40.216326 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:14:40.268030 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:40.413156 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:14:40.551407 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:40.566947 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:14:40.589358 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:14:40.752435 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:14:40.764523 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:40.969722 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:14:40.971271 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:14:40.972199 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:14:41.012611 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:41.259177 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:14:41.304999 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:41.510895 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:41.882838 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:41.922971 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:14:42.051443 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:42.125491 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:14:42.272567 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:14:42.280651 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:42.466434 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:42.547711 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:14:42.592705 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling FlushMRSOp(745d736d43244d4f9063f61de77cb473): perf score=0.033339
I20240503 08:14:42.614362 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling FlushMRSOp(ede24a27608a4cf196dd46f85c4163e1): perf score=0.033355
I20240503 08:14:42.677685 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling FlushMRSOp(ede24a27608a4cf196dd46f85c4163e1): perf score=0.033373
I20240503 08:14:42.726047 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: FlushMRSOp(ede24a27608a4cf196dd46f85c4163e1) complete. Timing: real 0.111s user 0.074s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":2609,"dirs.queue_time_us":6402,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":8486,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2006,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":16075,"threads_started":3,"wal-append.queue_time_us":6391}
I20240503 08:14:42.728396 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(ede24a27608a4cf196dd46f85c4163e1): 4923 bytes on disk
I20240503 08:14:42.729277 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(ede24a27608a4cf196dd46f85c4163e1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 08:14:42.730327 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling FlushMRSOp(3bfa8537bc8a4950b53fbf3c36d5ecf3): perf score=0.033380
I20240503 08:14:42.792862 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling FlushMRSOp(745d736d43244d4f9063f61de77cb473): perf score=0.033395
W20240503 08:14:42.796895 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:14:42.800109 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: FlushMRSOp(3bfa8537bc8a4950b53fbf3c36d5ecf3) complete. Timing: real 0.070s user 0.047s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1674,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":833,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":11773,"threads_started":1,"wal-append.queue_time_us":11868}
I20240503 08:14:42.801543 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(3bfa8537bc8a4950b53fbf3c36d5ecf3): 4838 bytes on disk
I20240503 08:14:42.802120 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(3bfa8537bc8a4950b53fbf3c36d5ecf3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240503 08:14:42.862948 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: FlushMRSOp(745d736d43244d4f9063f61de77cb473) complete. Timing: real 0.270s user 0.066s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":379,"dirs.queue_time_us":4407,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":2702,"drs_written":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4,"lbm_write_time_us":874,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":27255,"threads_started":3,"wal-append.queue_time_us":15183}
I20240503 08:14:42.872428 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(745d736d43244d4f9063f61de77cb473): 4583 bytes on disk
I20240503 08:14:42.873459 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(745d736d43244d4f9063f61de77cb473) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240503 08:14:42.898813 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: FlushMRSOp(ede24a27608a4cf196dd46f85c4163e1) complete. Timing: real 0.220s user 0.065s sys 0.025s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":13658,"dirs.queue_time_us":13129,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":908,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":979,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":48718,"threads_started":3,"wal-append.queue_time_us":21623}
I20240503 08:14:42.902287 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(ede24a27608a4cf196dd46f85c4163e1): 4923 bytes on disk
I20240503 08:14:42.903191 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(ede24a27608a4cf196dd46f85c4163e1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 08:14:43.022450 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: FlushMRSOp(745d736d43244d4f9063f61de77cb473) complete. Timing: real 0.229s user 0.075s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":6497,"dirs.queue_time_us":6262,"dirs.run_cpu_time_us":266,"dirs.run_wall_time_us":6307,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11158,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":28080,"threads_started":3,"wal-append.queue_time_us":15169}
I20240503 08:14:43.026377 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(745d736d43244d4f9063f61de77cb473): 4583 bytes on disk
I20240503 08:14:43.030465 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(745d736d43244d4f9063f61de77cb473) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240503 08:14:43.053406 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:14:43.092777 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:43.111446 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:14:43.222183 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:43.338972 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:14:43.392361 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:14:43.404969 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:14:43.541131 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:14:43.642478 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:43.681078 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:14:43.703866 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:14:43.908850 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling FlushMRSOp(b2ebb0b8475b4ed9a09711a87867de08): perf score=0.033344
I20240503 08:14:43.954861 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: FlushMRSOp(b2ebb0b8475b4ed9a09711a87867de08) complete. Timing: real 0.046s user 0.035s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":339,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1658,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":600,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":442,"threads_started":2,"wal-append.queue_time_us":306}
I20240503 08:14:43.955811 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(b2ebb0b8475b4ed9a09711a87867de08): 4838 bytes on disk
I20240503 08:14:43.956349 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(b2ebb0b8475b4ed9a09711a87867de08) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 08:14:44.058363 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling FlushMRSOp(0caf86850b3841ea811b0a6ab19ae330): perf score=0.033396
I20240503 08:14:44.108956 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: FlushMRSOp(0caf86850b3841ea811b0a6ab19ae330) complete. Timing: real 0.050s user 0.042s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":305,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1365,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":681,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":431,"threads_started":2,"wal-append.queue_time_us":347}
I20240503 08:14:44.109964 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling FlushMRSOp(c6e1b9f104ea471cab7527bd56433327): perf score=0.033400
I20240503 08:14:44.142277 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: FlushMRSOp(c6e1b9f104ea471cab7527bd56433327) complete. Timing: real 0.032s user 0.026s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":77,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1445,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":202,"threads_started":1,"wal-append.queue_time_us":308}
I20240503 08:14:44.143362 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(c6e1b9f104ea471cab7527bd56433327): 4921 bytes on disk
I20240503 08:14:44.143838 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(c6e1b9f104ea471cab7527bd56433327) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 08:14:44.144652 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(0caf86850b3841ea811b0a6ab19ae330): 4583 bytes on disk
I20240503 08:14:44.145078 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(0caf86850b3841ea811b0a6ab19ae330) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
W20240503 08:14:44.202417 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:44.292642 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:44.384269 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:14:44.659519 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:44.820627 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:14:44.882484 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling FlushMRSOp(e9997e8da99c4f45a63bc3db521b33ba): perf score=0.033347
I20240503 08:14:44.913656 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: FlushMRSOp(e9997e8da99c4f45a63bc3db521b33ba) complete. Timing: real 0.031s user 0.021s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":346,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1529,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":575,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":423,"threads_started":2,"wal-append.queue_time_us":310}
I20240503 08:14:44.914563 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(e9997e8da99c4f45a63bc3db521b33ba): 4923 bytes on disk
I20240503 08:14:44.915012 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(e9997e8da99c4f45a63bc3db521b33ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240503 08:14:44.915731 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling FlushMRSOp(ac392ff7965a405f96efc777aca4f0bb): perf score=0.033347
I20240503 08:14:44.960861 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling FlushMRSOp(d13c33b8712e4c488abb99b73ff42b86): perf score=0.033358
I20240503 08:14:44.967772 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: FlushMRSOp(ac392ff7965a405f96efc777aca4f0bb) complete. Timing: real 0.052s user 0.035s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":93,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1533,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":683,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1615,"threads_started":1,"wal-append.queue_time_us":1719}
I20240503 08:14:44.968770 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(ac392ff7965a405f96efc777aca4f0bb): 4838 bytes on disk
I20240503 08:14:44.969344 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(ac392ff7965a405f96efc777aca4f0bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240503 08:14:44.996013 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: FlushMRSOp(d13c33b8712e4c488abb99b73ff42b86) complete. Timing: real 0.035s user 0.022s sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":312,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1825,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":802,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":411,"threads_started":2,"wal-append.queue_time_us":283}
I20240503 08:14:44.997285 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(d13c33b8712e4c488abb99b73ff42b86): 4583 bytes on disk
W20240503 08:14:44.997768 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:14:44.997869 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(d13c33b8712e4c488abb99b73ff42b86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 08:14:44.998597 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling FlushMRSOp(e9997e8da99c4f45a63bc3db521b33ba): perf score=0.033366
I20240503 08:14:45.034802 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: FlushMRSOp(e9997e8da99c4f45a63bc3db521b33ba) complete. Timing: real 0.036s user 0.026s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1655,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":549,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":231,"threads_started":1,"wal-append.queue_time_us":339}
I20240503 08:14:45.036041 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(e9997e8da99c4f45a63bc3db521b33ba): 4923 bytes on disk
I20240503 08:14:45.036612 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(e9997e8da99c4f45a63bc3db521b33ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240503 08:14:45.040247 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling FlushMRSOp(d13c33b8712e4c488abb99b73ff42b86): perf score=0.033380
W20240503 08:14:45.062491 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:14:45.079910 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: FlushMRSOp(d13c33b8712e4c488abb99b73ff42b86) complete. Timing: real 0.039s user 0.029s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":328,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":1842,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":709,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":418,"threads_started":2,"wal-append.queue_time_us":295}
I20240503 08:14:45.081012 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(d13c33b8712e4c488abb99b73ff42b86): 4583 bytes on disk
I20240503 08:14:45.081506 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(d13c33b8712e4c488abb99b73ff42b86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240503 08:14:45.087870 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:14:45.088702 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:45.236320 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:14:45.351218 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:45.663548 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:14:45.673255 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:14:45.696130 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:45.713300 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:14:45.759043 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:14:45.832502 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:14:45.858580 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:45.871300 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:14:45.901388 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling FlushMRSOp(a8b4e398d34443b4a440f9ba48555d84): perf score=0.033345
I20240503 08:14:45.933871 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: FlushMRSOp(a8b4e398d34443b4a440f9ba48555d84) complete. Timing: real 0.032s user 0.023s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":321,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1718,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":602,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":476,"threads_started":2,"wal-append.queue_time_us":373}
I20240503 08:14:45.935004 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(a8b4e398d34443b4a440f9ba48555d84): 4838 bytes on disk
I20240503 08:14:45.935525 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(a8b4e398d34443b4a440f9ba48555d84) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 08:14:45.973336 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling FlushMRSOp(a8b4e398d34443b4a440f9ba48555d84): perf score=0.033368
I20240503 08:14:46.010243 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: FlushMRSOp(a8b4e398d34443b4a440f9ba48555d84) complete. Timing: real 0.037s user 0.035s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":301,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1844,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":779,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":440,"threads_started":2,"wal-append.queue_time_us":369}
I20240503 08:14:46.011284 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(a8b4e398d34443b4a440f9ba48555d84): 4838 bytes on disk
I20240503 08:14:46.011811 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(a8b4e398d34443b4a440f9ba48555d84) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 08:14:46.041304 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling FlushMRSOp(28e42c3244f949c7b7343bf587894ac5): perf score=0.033399
I20240503 08:14:46.086418 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling FlushMRSOp(d60a162155504975a39c20eff88341e1): perf score=0.033398
W20240503 08:14:46.093706 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:14:46.096134 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: FlushMRSOp(28e42c3244f949c7b7343bf587894ac5) complete. Timing: real 0.055s user 0.050s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":285,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1550,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":541,"threads_started":2,"wal-append.queue_time_us":486}
I20240503 08:14:46.097218 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling FlushMRSOp(d60a162155504975a39c20eff88341e1): perf score=0.033409
W20240503 08:14:46.126399 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:14:46.134900 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: FlushMRSOp(d60a162155504975a39c20eff88341e1) complete. Timing: real 0.037s user 0.031s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":83,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1719,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":613,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":240,"threads_started":1,"wal-append.queue_time_us":339}
I20240503 08:14:46.135962 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(d60a162155504975a39c20eff88341e1): 4921 bytes on disk
I20240503 08:14:46.136466 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(d60a162155504975a39c20eff88341e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240503 08:14:46.137297 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(28e42c3244f949c7b7343bf587894ac5): 4583 bytes on disk
I20240503 08:14:46.137702 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(28e42c3244f949c7b7343bf587894ac5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240503 08:14:46.138494 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: FlushMRSOp(d60a162155504975a39c20eff88341e1) complete. Timing: real 0.052s user 0.041s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":331,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1443,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":734,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":713,"threads_started":2,"wal-append.queue_time_us":301}
W20240503 08:14:46.138758 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:14:46.139834 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(d60a162155504975a39c20eff88341e1): 4921 bytes on disk
I20240503 08:14:46.140292 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(d60a162155504975a39c20eff88341e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 08:14:46.141208 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling FlushMRSOp(28e42c3244f949c7b7343bf587894ac5): perf score=0.033411
I20240503 08:14:46.171327 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: FlushMRSOp(28e42c3244f949c7b7343bf587894ac5) complete. Timing: real 0.030s user 0.015s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":70,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":2519,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":565,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":270,"threads_started":1,"wal-append.queue_time_us":395}
I20240503 08:14:46.172565 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(28e42c3244f949c7b7343bf587894ac5): 4583 bytes on disk
I20240503 08:14:46.173207 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(28e42c3244f949c7b7343bf587894ac5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240503 08:14:46.238343 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:46.564546 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:46.743064 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:46.786463 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:14:46.939574 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling FlushMRSOp(09156806b0744752969b6a574e24b50a): perf score=0.033335
I20240503 08:14:46.987116 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: FlushMRSOp(09156806b0744752969b6a574e24b50a) complete. Timing: real 0.047s user 0.037s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":345,"dirs.run_cpu_time_us":277,"dirs.run_wall_time_us":2193,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":724,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":476,"threads_started":2,"wal-append.queue_time_us":351}
I20240503 08:14:46.988112 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(09156806b0744752969b6a574e24b50a): 4838 bytes on disk
I20240503 08:14:46.988804 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(09156806b0744752969b6a574e24b50a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":177,"lbm_reads_lt_1ms":4}
I20240503 08:14:47.015741 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling FlushMRSOp(173376257c4946b5a801a27e4dffe90d): perf score=0.033351
W20240503 08:14:47.039722 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:14:47.059803 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: FlushMRSOp(173376257c4946b5a801a27e4dffe90d) complete. Timing: real 0.044s user 0.032s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":314,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1538,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":860,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":425,"threads_started":2,"wal-append.queue_time_us":295}
I20240503 08:14:47.061201 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling FlushMRSOp(b0d07a6278d545e69bf7d2012109d1f4): perf score=0.033360
I20240503 08:14:47.103968 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: FlushMRSOp(b0d07a6278d545e69bf7d2012109d1f4) complete. Timing: real 0.043s user 0.037s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":74,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1453,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":620,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":236,"threads_started":1,"wal-append.queue_time_us":449}
I20240503 08:14:47.105780 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(173376257c4946b5a801a27e4dffe90d): 4924 bytes on disk
I20240503 08:14:47.106518 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(173376257c4946b5a801a27e4dffe90d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
I20240503 08:14:47.107883 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(b0d07a6278d545e69bf7d2012109d1f4): 4584 bytes on disk
I20240503 08:14:47.108317 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(b0d07a6278d545e69bf7d2012109d1f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240503 08:14:47.141621 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling FlushMRSOp(173376257c4946b5a801a27e4dffe90d): perf score=0.033387
W20240503 08:14:47.176963 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:14:47.178084 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling FlushMRSOp(09156806b0744752969b6a574e24b50a): perf score=0.033393
I20240503 08:14:47.188331 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: FlushMRSOp(173376257c4946b5a801a27e4dffe90d) complete. Timing: real 0.046s user 0.035s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":361,"dirs.run_cpu_time_us":296,"dirs.run_wall_time_us":2144,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":668,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":446,"threads_started":2,"wal-append.queue_time_us":320}
I20240503 08:14:47.189519 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(173376257c4946b5a801a27e4dffe90d): 4924 bytes on disk
I20240503 08:14:47.190147 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(173376257c4946b5a801a27e4dffe90d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240503 08:14:47.215729 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: FlushMRSOp(09156806b0744752969b6a574e24b50a) complete. Timing: real 0.037s user 0.031s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":366,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":642,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":1280,"thread_start_us":576,"threads_started":2,"wal-append.queue_time_us":424}
I20240503 08:14:47.217209 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(09156806b0744752969b6a574e24b50a): 4838 bytes on disk
I20240503 08:14:47.217762 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(09156806b0744752969b6a574e24b50a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240503 08:14:47.333925 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:14:47.524546 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:14:47.541862 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:47.721846 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:47.947703 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:14:47.947752 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:47.951771 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:14:48.111605 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:48.362418 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:14:48.363265 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling FlushMRSOp(320d1e880958427a88e93321171cb36e): perf score=0.033349
W20240503 08:14:48.366694 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:14:48.391110 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:14:48.394315 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: FlushMRSOp(320d1e880958427a88e93321171cb36e) complete. Timing: real 0.031s user 0.017s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":391,"dirs.run_cpu_time_us":281,"dirs.run_wall_time_us":1809,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":679,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":502,"threads_started":2,"wal-append.queue_time_us":310}
I20240503 08:14:48.395377 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(320d1e880958427a88e93321171cb36e): 4583 bytes on disk
I20240503 08:14:48.395875 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(320d1e880958427a88e93321171cb36e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240503 08:14:48.400087 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:48.442414 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:14:48.444962 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling FlushMRSOp(d44335d8be7245b1bb52b2eab9e42c18): perf score=0.033382
I20240503 08:14:48.474470 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling FlushMRSOp(d44335d8be7245b1bb52b2eab9e42c18): perf score=0.033379
I20240503 08:14:48.477013 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: FlushMRSOp(d44335d8be7245b1bb52b2eab9e42c18) complete. Timing: real 0.032s user 0.031s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":343,"dirs.run_cpu_time_us":301,"dirs.run_wall_time_us":1947,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":589,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":586,"threads_started":2,"wal-append.queue_time_us":433}
I20240503 08:14:48.478160 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(d44335d8be7245b1bb52b2eab9e42c18): 4838 bytes on disk
I20240503 08:14:48.478920 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(d44335d8be7245b1bb52b2eab9e42c18) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240503 08:14:48.494932 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling FlushMRSOp(320d1e880958427a88e93321171cb36e): perf score=0.033395
W20240503 08:14:48.506506 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:14:48.512290 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: FlushMRSOp(d44335d8be7245b1bb52b2eab9e42c18) complete. Timing: real 0.038s user 0.035s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":409,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1543,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":612,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":508,"threads_started":2,"wal-append.queue_time_us":359}
I20240503 08:14:48.513697 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(d44335d8be7245b1bb52b2eab9e42c18): 4838 bytes on disk
I20240503 08:14:48.514257 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(d44335d8be7245b1bb52b2eab9e42c18) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240503 08:14:48.534967 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: FlushMRSOp(320d1e880958427a88e93321171cb36e) complete. Timing: real 0.040s user 0.026s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":343,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":1517,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":731,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":452,"threads_started":2,"wal-append.queue_time_us":319}
I20240503 08:14:48.536206 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(320d1e880958427a88e93321171cb36e): 4583 bytes on disk
I20240503 08:14:48.536739 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(320d1e880958427a88e93321171cb36e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240503 08:14:48.537699 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling FlushMRSOp(4babf0ea9b954a23b1b1c4e403dce0bd): perf score=0.033401
I20240503 08:14:48.576385 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: FlushMRSOp(4babf0ea9b954a23b1b1c4e403dce0bd) complete. Timing: real 0.038s user 0.032s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":2717,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":804,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":245,"threads_started":1,"wal-append.queue_time_us":413}
I20240503 08:14:48.577736 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(4babf0ea9b954a23b1b1c4e403dce0bd): 4923 bytes on disk
I20240503 08:14:48.578302 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(4babf0ea9b954a23b1b1c4e403dce0bd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
W20240503 08:14:48.699807 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:48.769160 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:14:48.843515 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:49.143765 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:49.174518 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:14:49.232084 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:14:49.399709 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling FlushMRSOp(a20489bf65774902af38d228ee966ffe): perf score=0.033335
I20240503 08:14:49.433533 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: FlushMRSOp(a20489bf65774902af38d228ee966ffe) complete. Timing: real 0.033s user 0.027s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":352,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":1519,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":766,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":443,"threads_started":2,"wal-append.queue_time_us":326}
I20240503 08:14:49.434531 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(a20489bf65774902af38d228ee966ffe): 4838 bytes on disk
I20240503 08:14:49.435045 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(a20489bf65774902af38d228ee966ffe) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240503 08:14:49.519681 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling FlushMRSOp(585ba864cc384e7f9db90e9d4c47e82c): perf score=0.033373
I20240503 08:14:49.565143 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: FlushMRSOp(585ba864cc384e7f9db90e9d4c47e82c) complete. Timing: real 0.045s user 0.034s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":348,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":2152,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1191,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":517,"threads_started":2,"wal-append.queue_time_us":391}
I20240503 08:14:49.566188 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(585ba864cc384e7f9db90e9d4c47e82c): 4583 bytes on disk
I20240503 08:14:49.566686 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(585ba864cc384e7f9db90e9d4c47e82c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240503 08:14:49.582141 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling FlushMRSOp(9b3b8b80ebd44f27899f97211266c7b2): perf score=0.033384
I20240503 08:14:49.645701 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: FlushMRSOp(9b3b8b80ebd44f27899f97211266c7b2) complete. Timing: real 0.063s user 0.054s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":423,"dirs.run_cpu_time_us":311,"dirs.run_wall_time_us":1840,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":784,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":540,"threads_started":2,"wal-append.queue_time_us":357}
I20240503 08:14:49.646754 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(9b3b8b80ebd44f27899f97211266c7b2): 4923 bytes on disk
I20240503 08:14:49.647264 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(9b3b8b80ebd44f27899f97211266c7b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240503 08:14:49.667356 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:49.780355 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:49.884733 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:14:50.120064 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:14:50.190958 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:50.242113 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240503 08:14:50.384585 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:50.466223 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:14:50.579170 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:14:50.651465 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling FlushMRSOp(564fb1b51abd4f8da0372ec99e3c599c): perf score=0.033343
W20240503 08:14:50.665048 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:14:50.681833 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: FlushMRSOp(564fb1b51abd4f8da0372ec99e3c599c) complete. Timing: real 0.030s user 0.025s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":339,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":1859,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":690,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":461,"threads_started":2,"wal-append.queue_time_us":335}
I20240503 08:14:50.683032 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling FlushMRSOp(b5736ee8065344ea8124fdf759035216): perf score=0.033348
I20240503 08:14:50.689467 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling FlushMRSOp(b5736ee8065344ea8124fdf759035216): perf score=0.033348
W20240503 08:14:50.709760 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:14:50.715695 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: FlushMRSOp(b5736ee8065344ea8124fdf759035216) complete. Timing: real 0.032s user 0.015s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1623,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":742,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":239,"threads_started":1,"wal-append.queue_time_us":354}
I20240503 08:14:50.717209 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(b5736ee8065344ea8124fdf759035216): 4838 bytes on disk
I20240503 08:14:50.717908 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(b5736ee8065344ea8124fdf759035216) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240503 08:14:50.719195 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(564fb1b51abd4f8da0372ec99e3c599c): 4581 bytes on disk
I20240503 08:14:50.719672 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(564fb1b51abd4f8da0372ec99e3c599c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 08:14:50.723414 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: FlushMRSOp(b5736ee8065344ea8124fdf759035216) complete. Timing: real 0.033s user 0.026s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":332,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1521,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":756,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":489,"threads_started":2,"wal-append.queue_time_us":346}
I20240503 08:14:50.724450 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(b5736ee8065344ea8124fdf759035216): 4838 bytes on disk
I20240503 08:14:50.725023 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(b5736ee8065344ea8124fdf759035216) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240503 08:14:50.737536 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling FlushMRSOp(564fb1b51abd4f8da0372ec99e3c599c): perf score=0.033361
I20240503 08:14:50.768520 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: FlushMRSOp(564fb1b51abd4f8da0372ec99e3c599c) complete. Timing: real 0.031s user 0.024s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":379,"dirs.run_cpu_time_us":298,"dirs.run_wall_time_us":1892,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":688,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":466,"threads_started":2,"wal-append.queue_time_us":348}
I20240503 08:14:50.769654 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(564fb1b51abd4f8da0372ec99e3c599c): 4581 bytes on disk
I20240503 08:14:50.770350 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(564fb1b51abd4f8da0372ec99e3c599c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
W20240503 08:14:50.814944 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:14:50.822788 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling FlushMRSOp(b4ceeeb3495e4a8fac6e13e431c64af4): perf score=0.033391
W20240503 08:14:50.843552 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:14:50.867053 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: FlushMRSOp(b4ceeeb3495e4a8fac6e13e431c64af4) complete. Timing: real 0.044s user 0.031s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":344,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":1918,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":891,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":486,"threads_started":2,"wal-append.queue_time_us":385}
I20240503 08:14:50.868155 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(b4ceeeb3495e4a8fac6e13e431c64af4): 4923 bytes on disk
I20240503 08:14:50.868674 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(b4ceeeb3495e4a8fac6e13e431c64af4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240503 08:14:51.018179 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:14:51.041996 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:51.096840 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:14:51.187072 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:14:51.284072 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:14:51.427745 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:51.527565 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:51.572965 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:14:51.594992 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:14:51.873071 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:52.071796 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:14:52.124666 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling FlushMRSOp(2d113e8de02f447b9da0a1cb03686cd2): perf score=0.033344
I20240503 08:14:52.160848 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: FlushMRSOp(2d113e8de02f447b9da0a1cb03686cd2) complete. Timing: real 0.036s user 0.020s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":290,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1667,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":644,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":425,"threads_started":2,"wal-append.queue_time_us":316}
I20240503 08:14:52.162189 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(2d113e8de02f447b9da0a1cb03686cd2): 4838 bytes on disk
I20240503 08:14:52.162832 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(2d113e8de02f447b9da0a1cb03686cd2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240503 08:14:52.163898 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling FlushMRSOp(ba833ce23c4c4e81bc1a81125a1208c9): perf score=0.033351
I20240503 08:14:52.214246 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: FlushMRSOp(ba833ce23c4c4e81bc1a81125a1208c9) complete. Timing: real 0.050s user 0.027s sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1972,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":554,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":235,"threads_started":1,"wal-append.queue_time_us":354}
I20240503 08:14:52.215607 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(ba833ce23c4c4e81bc1a81125a1208c9): 4923 bytes on disk
I20240503 08:14:52.216138 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(ba833ce23c4c4e81bc1a81125a1208c9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240503 08:14:52.225840 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling FlushMRSOp(a8a9adc3712d436c95505a3216e7607c): perf score=0.033375
I20240503 08:14:52.230720 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling FlushMRSOp(ba833ce23c4c4e81bc1a81125a1208c9): perf score=0.033370
I20240503 08:14:52.276818 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling FlushMRSOp(ba833ce23c4c4e81bc1a81125a1208c9): perf score=0.033384
I20240503 08:14:52.289661 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: FlushMRSOp(ba833ce23c4c4e81bc1a81125a1208c9) complete. Timing: real 0.059s user 0.033s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1167,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":2898,"drs_written":1,"lbm_read_time_us":258,"lbm_reads_lt_1ms":4,"lbm_write_time_us":681,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3152,"threads_started":2,"wal-append.queue_time_us":403}
I20240503 08:14:52.290755 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(ba833ce23c4c4e81bc1a81125a1208c9): 4923 bytes on disk
I20240503 08:14:52.291332 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(ba833ce23c4c4e81bc1a81125a1208c9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240503 08:14:52.295053 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: FlushMRSOp(a8a9adc3712d436c95505a3216e7607c) complete. Timing: real 0.069s user 0.048s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":7651,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":2857,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":967,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7816,"threads_started":2,"wal-append.queue_time_us":388}
I20240503 08:14:52.296087 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(a8a9adc3712d436c95505a3216e7607c): 4583 bytes on disk
I20240503 08:14:52.296666 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(a8a9adc3712d436c95505a3216e7607c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240503 08:14:52.330111 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:14:52.334770 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: FlushMRSOp(ba833ce23c4c4e81bc1a81125a1208c9) complete. Timing: real 0.058s user 0.043s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":353,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1540,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":667,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":489,"threads_started":2,"wal-append.queue_time_us":328}
I20240503 08:14:52.335948 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling FlushMRSOp(a8a9adc3712d436c95505a3216e7607c): perf score=0.033398
I20240503 08:14:52.374401 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: FlushMRSOp(a8a9adc3712d436c95505a3216e7607c) complete. Timing: real 0.038s user 0.029s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1567,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":592,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":250,"threads_started":1,"wal-append.queue_time_us":375}
I20240503 08:14:52.375856 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(ba833ce23c4c4e81bc1a81125a1208c9): 4923 bytes on disk
I20240503 08:14:52.376462 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(ba833ce23c4c4e81bc1a81125a1208c9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240503 08:14:52.377444 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(a8a9adc3712d436c95505a3216e7607c): 4583 bytes on disk
I20240503 08:14:52.378208 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(a8a9adc3712d436c95505a3216e7607c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240503 08:14:52.517732 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:14:52.663940 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:52.742945 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:14:52.783006 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:52.991919 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:14:53.089359 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:53.125418 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:14:53.268041 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:14:53.368705 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:53.473193 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:14:53.546128 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:14:53.607292 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:53.700208 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:14:53.710050 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:14:53.938134 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:14:53.992911 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:54.051874 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:14:54.099004 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:14:54.224422 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling FlushMRSOp(6e9537f30aaf45adbc2cc4a784009de7): perf score=0.033352
I20240503 08:14:54.270305 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: FlushMRSOp(6e9537f30aaf45adbc2cc4a784009de7) complete. Timing: real 0.046s user 0.031s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":381,"dirs.run_cpu_time_us":276,"dirs.run_wall_time_us":1865,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":831,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":762,"threads_started":2,"wal-append.queue_time_us":621}
I20240503 08:14:54.271948 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(6e9537f30aaf45adbc2cc4a784009de7): 4923 bytes on disk
I20240503 08:14:54.272626 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(6e9537f30aaf45adbc2cc4a784009de7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
I20240503 08:14:54.273957 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling FlushMRSOp(da529c8f8de241d38678d1596b9402ac): perf score=0.033356
I20240503 08:14:54.298919 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling FlushMRSOp(da529c8f8de241d38678d1596b9402ac): perf score=0.033359
I20240503 08:14:54.304111 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling FlushMRSOp(6e9537f30aaf45adbc2cc4a784009de7): perf score=0.033365
I20240503 08:14:54.335098 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: FlushMRSOp(da529c8f8de241d38678d1596b9402ac) complete. Timing: real 0.061s user 0.042s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4512,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":1634,"drs_written":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4,"lbm_write_time_us":635,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11177,"threads_started":1,"wal-append.queue_time_us":11281}
I20240503 08:14:54.336645 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(da529c8f8de241d38678d1596b9402ac): 4583 bytes on disk
I20240503 08:14:54.337282 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(da529c8f8de241d38678d1596b9402ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240503 08:14:54.356642 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: FlushMRSOp(da529c8f8de241d38678d1596b9402ac) complete. Timing: real 0.057s user 0.036s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4793,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1478,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":744,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":4958,"threads_started":2,"wal-append.queue_time_us":367}
I20240503 08:14:54.357734 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(da529c8f8de241d38678d1596b9402ac): 4583 bytes on disk
I20240503 08:14:54.358364 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(da529c8f8de241d38678d1596b9402ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240503 08:14:54.365799 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: FlushMRSOp(6e9537f30aaf45adbc2cc4a784009de7) complete. Timing: real 0.061s user 0.043s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":386,"dirs.run_cpu_time_us":284,"dirs.run_wall_time_us":1626,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1031,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":571,"threads_started":2,"wal-append.queue_time_us":420}
I20240503 08:14:54.367091 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(6e9537f30aaf45adbc2cc4a784009de7): 4923 bytes on disk
I20240503 08:14:54.367695 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(6e9537f30aaf45adbc2cc4a784009de7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20240503 08:14:54.378252 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:14:54.386396 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling FlushMRSOp(48e31346107a453a81921b23b244e7ff): perf score=0.033392
I20240503 08:14:54.438063 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: FlushMRSOp(48e31346107a453a81921b23b244e7ff) complete. Timing: real 0.051s user 0.045s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":692,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1626,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":960,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":788,"threads_started":2,"wal-append.queue_time_us":289}
I20240503 08:14:54.439169 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(48e31346107a453a81921b23b244e7ff): 4838 bytes on disk
I20240503 08:14:54.439741 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(48e31346107a453a81921b23b244e7ff) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20240503 08:14:54.712618 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:54.833456 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:55.121547 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:14:55.148193 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:14:55.227661 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:55.257666 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:14:55.313158 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:55.448064 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:14:55.643018 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:14:55.787626 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:55.834236 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:14:55.842350 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:55.899354 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:14:55.914537 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:14:55.930486 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:14:56.147578 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:14:56.199478 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:14:56.400691 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:56.462286 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:56.508598 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:14:56.610443 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:14:56.788295 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:57.075181 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:57.392927 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:57.439461 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:14:57.547663 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:57.904162 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:57.931674 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:14:57.932698 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:14:57.937456 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:58.125494 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:14:58.279130 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:14:58.416265 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:14:58.451370 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:58.500679 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:14:58.514678 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:14:58.525583 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:58.530805 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:14:58.611779 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:14:58.987439 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:58.997649 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:14:59.057468 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:59.508074 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:59.539695 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:14:59.943121 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:14:59.986707 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:00.069770 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:00.149082 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:15:00.273425 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:15:00.429538 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:15:00.514950 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:00.671329 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:00.676342 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:15:00.725939 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:15:00.735909 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:15:00.842347 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:15:00.887506 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:15:00.907895 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:15:00.916626 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:01.147195 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:01.248769 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:15:01.381165 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:01.441731 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:15:01.552891 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:01.851671 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:01.870985 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 08:15:01.978646 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:02.410686 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:02.475246 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:02.518178 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:15:02.554832 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:15:02.814591 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:03.044668 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:15:03.062944 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:15:03.073105 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:03.088689 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:15:03.174082 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:15:03.212687 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:03.234366 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:15:03.427320 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:15:03.432202 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:15:03.565346 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:03.675536 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:03.943840 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:15:03.948733 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:04.211902 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:04.231886 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:15:04.347730 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:04.774811 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:04.866492 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:04.913462 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:15:05.031657 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:15:05.150130 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:15:05.150945 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:05.340133 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:05.377067 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:15:05.640123 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:15:05.729692 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:05.731586 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:15:05.739115 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:15:05.760843 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:15:05.830885 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:15:05.848716 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling FlushMRSOp(b2ebb0b8475b4ed9a09711a87867de08): perf score=0.033373
W20240503 08:15:05.892390 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:05.920240 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: FlushMRSOp(b2ebb0b8475b4ed9a09711a87867de08) complete. Timing: real 0.071s user 0.049s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":428,"dirs.queue_time_us":3816,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1668,"drs_written":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4,"lbm_write_time_us":837,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":4425,"threads_started":3,"wal-append.queue_time_us":396}
I20240503 08:15:05.922785 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling UndoDeltaBlockGCOp(b2ebb0b8475b4ed9a09711a87867de08): 4838 bytes on disk
I20240503 08:15:05.930060 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: UndoDeltaBlockGCOp(b2ebb0b8475b4ed9a09711a87867de08) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":158,"lbm_reads_lt_1ms":4}
I20240503 08:15:05.932161 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling FlushMRSOp(c6e1b9f104ea471cab7527bd56433327): perf score=0.033391
I20240503 08:15:06.011510 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: FlushMRSOp(c6e1b9f104ea471cab7527bd56433327) complete. Timing: real 0.079s user 0.057s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":2035,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1476,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":257,"threads_started":1,"wal-append.queue_time_us":389}
I20240503 08:15:06.013067 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling UndoDeltaBlockGCOp(c6e1b9f104ea471cab7527bd56433327): 4921 bytes on disk
I20240503 08:15:06.013754 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: UndoDeltaBlockGCOp(c6e1b9f104ea471cab7527bd56433327) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240503 08:15:06.014680 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling FlushMRSOp(6e9537f30aaf45adbc2cc4a784009de7): perf score=0.033410
W20240503 08:15:06.039835 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:15:06.067626 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: FlushMRSOp(6e9537f30aaf45adbc2cc4a784009de7) complete. Timing: real 0.053s user 0.036s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":74,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1899,"drs_written":1,"lbm_read_time_us":478,"lbm_reads_lt_1ms":4,"lbm_write_time_us":972,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":332,"threads_started":1,"wal-append.queue_time_us":497}
I20240503 08:15:06.069088 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling UndoDeltaBlockGCOp(6e9537f30aaf45adbc2cc4a784009de7): 4923 bytes on disk
I20240503 08:15:06.069710 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: UndoDeltaBlockGCOp(6e9537f30aaf45adbc2cc4a784009de7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240503 08:15:06.200683 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:06.294306 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:06.321863 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling FlushMRSOp(585ba864cc384e7f9db90e9d4c47e82c): perf score=0.033395
I20240503 08:15:06.379171 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling FlushMRSOp(320d1e880958427a88e93321171cb36e): perf score=0.033382
I20240503 08:15:06.380538 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling FlushMRSOp(b2ebb0b8475b4ed9a09711a87867de08): perf score=0.033362
I20240503 08:15:06.381923 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: FlushMRSOp(585ba864cc384e7f9db90e9d4c47e82c) complete. Timing: real 0.060s user 0.056s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1852,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":912,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":261,"threads_started":1,"wal-append.queue_time_us":405}
I20240503 08:15:06.384081 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling UndoDeltaBlockGCOp(585ba864cc384e7f9db90e9d4c47e82c): 4583 bytes on disk
I20240503 08:15:06.384768 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: UndoDeltaBlockGCOp(585ba864cc384e7f9db90e9d4c47e82c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240503 08:15:06.537403 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: FlushMRSOp(320d1e880958427a88e93321171cb36e) complete. Timing: real 0.157s user 0.059s sys 0.021s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":1317,"dirs.queue_time_us":10171,"dirs.run_cpu_time_us":301,"dirs.run_wall_time_us":7513,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1068,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":23262,"threads_started":3,"wal-append.queue_time_us":6117}
I20240503 08:15:06.537217 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: FlushMRSOp(b2ebb0b8475b4ed9a09711a87867de08) complete. Timing: real 0.156s user 0.077s sys 0.026s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":14789,"dirs.queue_time_us":6391,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":4035,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":939,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":39224,"threads_started":3,"wal-append.queue_time_us":13555}
I20240503 08:15:06.539434 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling UndoDeltaBlockGCOp(320d1e880958427a88e93321171cb36e): 4583 bytes on disk
I20240503 08:15:06.539683 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling FlushMRSOp(48e31346107a453a81921b23b244e7ff): perf score=0.033388
I20240503 08:15:06.540308 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: UndoDeltaBlockGCOp(320d1e880958427a88e93321171cb36e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240503 08:15:06.541473 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling FlushMRSOp(3bfa8537bc8a4950b53fbf3c36d5ecf3): perf score=0.033385
I20240503 08:15:06.657954 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: FlushMRSOp(3bfa8537bc8a4950b53fbf3c36d5ecf3) complete. Timing: real 0.116s user 0.054s sys 0.021s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1469,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":962,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":16400,"threads_started":1,"wal-append.queue_time_us":16564}
I20240503 08:15:06.659459 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling UndoDeltaBlockGCOp(3bfa8537bc8a4950b53fbf3c36d5ecf3): 4838 bytes on disk
I20240503 08:15:06.665601 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: UndoDeltaBlockGCOp(3bfa8537bc8a4950b53fbf3c36d5ecf3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240503 08:15:06.666973 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling FlushMRSOp(b0d07a6278d545e69bf7d2012109d1f4): perf score=0.033418
I20240503 08:15:06.695158 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: FlushMRSOp(48e31346107a453a81921b23b244e7ff) complete. Timing: real 0.155s user 0.053s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":123,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1519,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5961,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":896,"thread_start_us":10664,"threads_started":1,"wal-append.queue_time_us":7658}
I20240503 08:15:06.697270 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling UndoDeltaBlockGCOp(48e31346107a453a81921b23b244e7ff): 4838 bytes on disk
I20240503 08:15:06.698354 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: UndoDeltaBlockGCOp(48e31346107a453a81921b23b244e7ff) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
I20240503 08:15:06.700032 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling FlushMRSOp(0caf86850b3841ea811b0a6ab19ae330): perf score=0.033411
W20240503 08:15:06.756438 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:15:06.770112 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: FlushMRSOp(b0d07a6278d545e69bf7d2012109d1f4) complete. Timing: real 0.095s user 0.051s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":72,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":2056,"drs_written":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1049,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":10059,"threads_started":1,"wal-append.queue_time_us":6094}
I20240503 08:15:06.771364 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: FlushMRSOp(0caf86850b3841ea811b0a6ab19ae330) complete. Timing: real 0.071s user 0.047s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":64,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1495,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1014,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3494,"threads_started":1,"wal-append.queue_time_us":3371}
I20240503 08:15:06.771740 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling UndoDeltaBlockGCOp(b0d07a6278d545e69bf7d2012109d1f4): 4584 bytes on disk
I20240503 08:15:06.772336 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: UndoDeltaBlockGCOp(b0d07a6278d545e69bf7d2012109d1f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240503 08:15:06.772897 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling FlushMRSOp(3bfa8537bc8a4950b53fbf3c36d5ecf3): perf score=0.033383
I20240503 08:15:06.773453 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling FlushMRSOp(ac392ff7965a405f96efc777aca4f0bb): perf score=0.033369
W20240503 08:15:06.775197 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:15:06.789868 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:06.882079 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:06.884107 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:15:06.888376 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling FlushMRSOp(b0d07a6278d545e69bf7d2012109d1f4): perf score=0.033371
I20240503 08:15:06.891700 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: FlushMRSOp(3bfa8537bc8a4950b53fbf3c36d5ecf3) complete. Timing: real 0.118s user 0.048s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3481,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":1098,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":820,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":4480,"thread_start_us":21406,"threads_started":1,"wal-append.queue_time_us":21286}
I20240503 08:15:06.893272 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling UndoDeltaBlockGCOp(3bfa8537bc8a4950b53fbf3c36d5ecf3): 4838 bytes on disk
I20240503 08:15:06.893971 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: UndoDeltaBlockGCOp(3bfa8537bc8a4950b53fbf3c36d5ecf3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
I20240503 08:15:06.895193 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling UndoDeltaBlockGCOp(b2ebb0b8475b4ed9a09711a87867de08): 4838 bytes on disk
I20240503 08:15:06.895781 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: UndoDeltaBlockGCOp(b2ebb0b8475b4ed9a09711a87867de08) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240503 08:15:06.896936 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling UndoDeltaBlockGCOp(0caf86850b3841ea811b0a6ab19ae330): 4583 bytes on disk
I20240503 08:15:06.897452 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: UndoDeltaBlockGCOp(0caf86850b3841ea811b0a6ab19ae330) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240503 08:15:06.929554 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: FlushMRSOp(ac392ff7965a405f96efc777aca4f0bb) complete. Timing: real 0.156s user 0.044s sys 0.022s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":1376,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1190,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":23067,"threads_started":1,"wal-append.queue_time_us":22966}
I20240503 08:15:06.931120 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling UndoDeltaBlockGCOp(ac392ff7965a405f96efc777aca4f0bb): 4838 bytes on disk
I20240503 08:15:06.931849 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: UndoDeltaBlockGCOp(ac392ff7965a405f96efc777aca4f0bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20240503 08:15:06.947319 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: FlushMRSOp(b0d07a6278d545e69bf7d2012109d1f4) complete. Timing: real 0.059s user 0.037s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6306,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":5934,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":791,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6685,"threads_started":2,"wal-append.queue_time_us":361}
I20240503 08:15:06.948767 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling UndoDeltaBlockGCOp(b0d07a6278d545e69bf7d2012109d1f4): 4584 bytes on disk
I20240503 08:15:06.949440 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: UndoDeltaBlockGCOp(b0d07a6278d545e69bf7d2012109d1f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240503 08:15:07.149606 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling FlushMRSOp(4babf0ea9b954a23b1b1c4e403dce0bd): perf score=0.033346
I20240503 08:15:07.214561 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: FlushMRSOp(4babf0ea9b954a23b1b1c4e403dce0bd) complete. Timing: real 0.065s user 0.046s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":68,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1670,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":958,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1289,"threads_started":1,"wal-append.queue_time_us":427}
I20240503 08:15:07.216041 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling UndoDeltaBlockGCOp(4babf0ea9b954a23b1b1c4e403dce0bd): 4923 bytes on disk
I20240503 08:15:07.216703 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: UndoDeltaBlockGCOp(4babf0ea9b954a23b1b1c4e403dce0bd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
W20240503 08:15:07.342290 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:07.412837 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:15:07.442991 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:07.473955 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:15:07.926116 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:15:07.931496 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:07.945484 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:08.017976 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:15:08.036182 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:15:08.268952 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:15:08.380538 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:15:08.389694 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:08.455570 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling FlushMRSOp(745d736d43244d4f9063f61de77cb473): perf score=0.033403
I20240503 08:15:08.494057 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: FlushMRSOp(745d736d43244d4f9063f61de77cb473) complete. Timing: real 0.038s user 0.025s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":382,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":2061,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":742,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":513,"threads_started":2,"wal-append.queue_time_us":378}
I20240503 08:15:08.495186 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling UndoDeltaBlockGCOp(745d736d43244d4f9063f61de77cb473): 4583 bytes on disk
I20240503 08:15:08.495743 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: UndoDeltaBlockGCOp(745d736d43244d4f9063f61de77cb473) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240503 08:15:08.530449 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:08.601017 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:15:08.620560 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:15:08.688726 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling FlushMRSOp(09156806b0744752969b6a574e24b50a): perf score=0.033370
I20240503 08:15:08.728508 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: FlushMRSOp(09156806b0744752969b6a574e24b50a) complete. Timing: real 0.039s user 0.034s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":301,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1924,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":632,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":406,"threads_started":2,"wal-append.queue_time_us":330}
I20240503 08:15:08.729771 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling UndoDeltaBlockGCOp(09156806b0744752969b6a574e24b50a): 4838 bytes on disk
I20240503 08:15:08.730293 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: UndoDeltaBlockGCOp(09156806b0744752969b6a574e24b50a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240503 08:15:08.833847 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:09.060179 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:09.276758 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:15:09.317158 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:15:09.456058 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:09.483690 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling FlushMRSOp(c6e1b9f104ea471cab7527bd56433327): perf score=0.033349
W20240503 08:15:09.484910 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:09.523905 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: FlushMRSOp(c6e1b9f104ea471cab7527bd56433327) complete. Timing: real 0.040s user 0.033s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":346,"dirs.run_cpu_time_us":291,"dirs.run_wall_time_us":1773,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":767,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3208,"threads_started":2,"wal-append.queue_time_us":3077}
I20240503 08:15:09.525077 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling UndoDeltaBlockGCOp(c6e1b9f104ea471cab7527bd56433327): 4921 bytes on disk
I20240503 08:15:09.526810 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: UndoDeltaBlockGCOp(c6e1b9f104ea471cab7527bd56433327) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240503 08:15:09.613562 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:15:09.969293 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:09.979447 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling FlushMRSOp(2d113e8de02f447b9da0a1cb03686cd2): perf score=0.033336
I20240503 08:15:10.015097 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: FlushMRSOp(2d113e8de02f447b9da0a1cb03686cd2) complete. Timing: real 0.035s user 0.006s sys 0.028s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":343,"dirs.run_cpu_time_us":278,"dirs.run_wall_time_us":1607,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":723,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":556,"threads_started":2,"wal-append.queue_time_us":407}
I20240503 08:15:10.016551 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling UndoDeltaBlockGCOp(2d113e8de02f447b9da0a1cb03686cd2): 4838 bytes on disk
I20240503 08:15:10.017197 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: UndoDeltaBlockGCOp(2d113e8de02f447b9da0a1cb03686cd2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":147,"lbm_reads_lt_1ms":4}
W20240503 08:15:10.019881 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:15:10.047597 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:10.195115 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 08:15:10.280972 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling FlushMRSOp(d60a162155504975a39c20eff88341e1): perf score=0.033389
I20240503 08:15:10.323707 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: FlushMRSOp(d60a162155504975a39c20eff88341e1) complete. Timing: real 0.042s user 0.025s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":366,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1483,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1508,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":458,"threads_started":2,"wal-append.queue_time_us":325}
I20240503 08:15:10.325295 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling UndoDeltaBlockGCOp(d60a162155504975a39c20eff88341e1): 4921 bytes on disk
I20240503 08:15:10.325918 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: UndoDeltaBlockGCOp(d60a162155504975a39c20eff88341e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240503 08:15:10.425696 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:15:10.426465 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:15:10.430509 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:15:10.436275 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:15:10.520323 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling FlushMRSOp(28e42c3244f949c7b7343bf587894ac5): perf score=0.033334
W20240503 08:15:10.550390 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:10.560684 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: FlushMRSOp(28e42c3244f949c7b7343bf587894ac5) complete. Timing: real 0.040s user 0.030s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":310,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":2302,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":767,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":447,"threads_started":2,"wal-append.queue_time_us":334}
I20240503 08:15:10.562105 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling UndoDeltaBlockGCOp(28e42c3244f949c7b7343bf587894ac5): 4583 bytes on disk
I20240503 08:15:10.562893 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: UndoDeltaBlockGCOp(28e42c3244f949c7b7343bf587894ac5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
W20240503 08:15:10.622668 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:10.693042 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:15:10.755676 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling FlushMRSOp(d44335d8be7245b1bb52b2eab9e42c18): perf score=0.033384
I20240503 08:15:10.789458 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: FlushMRSOp(d44335d8be7245b1bb52b2eab9e42c18) complete. Timing: real 0.034s user 0.017s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":384,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":1560,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":629,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":597,"threads_started":2,"wal-append.queue_time_us":449}
I20240503 08:15:10.790849 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling UndoDeltaBlockGCOp(d44335d8be7245b1bb52b2eab9e42c18): 4838 bytes on disk
I20240503 08:15:10.791388 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: UndoDeltaBlockGCOp(d44335d8be7245b1bb52b2eab9e42c18) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240503 08:15:10.896023 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:15:10.993410 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:15:11.082409 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:11.159410 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:11.331379 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling FlushMRSOp(9b3b8b80ebd44f27899f97211266c7b2): perf score=0.033382
I20240503 08:15:11.372802 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: FlushMRSOp(9b3b8b80ebd44f27899f97211266c7b2) complete. Timing: real 0.041s user 0.034s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":330,"dirs.run_cpu_time_us":276,"dirs.run_wall_time_us":1799,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":589,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":485,"threads_started":2,"wal-append.queue_time_us":363}
I20240503 08:15:11.374145 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling UndoDeltaBlockGCOp(9b3b8b80ebd44f27899f97211266c7b2): 4923 bytes on disk
I20240503 08:15:11.374683 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: UndoDeltaBlockGCOp(9b3b8b80ebd44f27899f97211266c7b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240503 08:15:11.545056 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling FlushMRSOp(a8a9adc3712d436c95505a3216e7607c): perf score=0.033340
I20240503 08:15:11.584770 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: FlushMRSOp(a8a9adc3712d436c95505a3216e7607c) complete. Timing: real 0.039s user 0.029s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":363,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":2380,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":847,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1207,"threads_started":2,"wal-append.queue_time_us":1038}
I20240503 08:15:11.586208 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling UndoDeltaBlockGCOp(a8a9adc3712d436c95505a3216e7607c): 4583 bytes on disk
I20240503 08:15:11.586692 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: UndoDeltaBlockGCOp(a8a9adc3712d436c95505a3216e7607c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240503 08:15:11.641654 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:11.663278 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:15:11.667578 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:11.698524 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:15:11.838474 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling FlushMRSOp(a8b4e398d34443b4a440f9ba48555d84): perf score=0.033403
I20240503 08:15:11.890072 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: FlushMRSOp(a8b4e398d34443b4a440f9ba48555d84) complete. Timing: real 0.051s user 0.037s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":93,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":1704,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":601,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":192,"threads_started":1,"wal-append.queue_time_us":301}
I20240503 08:15:11.891147 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling UndoDeltaBlockGCOp(a8b4e398d34443b4a440f9ba48555d84): 4838 bytes on disk
I20240503 08:15:11.891645 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: UndoDeltaBlockGCOp(a8b4e398d34443b4a440f9ba48555d84) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240503 08:15:11.915051 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:15:12.145273 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:12.154189 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:12.219121 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:15:12.542541 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:12.602254 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:12.762079 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:15:12.782806 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:15:12.846287 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:15:12.949530 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:15:13.077450 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling FlushMRSOp(9b3b8b80ebd44f27899f97211266c7b2): perf score=0.033339
I20240503 08:15:13.119633 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: FlushMRSOp(9b3b8b80ebd44f27899f97211266c7b2) complete. Timing: real 0.042s user 0.031s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":347,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1543,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":741,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":492,"threads_started":2,"wal-append.queue_time_us":380}
I20240503 08:15:13.120610 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling UndoDeltaBlockGCOp(9b3b8b80ebd44f27899f97211266c7b2): 4923 bytes on disk
I20240503 08:15:13.121132 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: UndoDeltaBlockGCOp(9b3b8b80ebd44f27899f97211266c7b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
W20240503 08:15:13.122891 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:13.191888 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:13.279848 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:15:13.381719 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling FlushMRSOp(da529c8f8de241d38678d1596b9402ac): perf score=0.033377
W20240503 08:15:13.406323 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:15:13.411664 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: FlushMRSOp(da529c8f8de241d38678d1596b9402ac) complete. Timing: real 0.030s user 0.016s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":346,"dirs.run_cpu_time_us":271,"dirs.run_wall_time_us":1758,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":596,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":496,"threads_started":2,"wal-append.queue_time_us":336}
I20240503 08:15:13.412715 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling UndoDeltaBlockGCOp(da529c8f8de241d38678d1596b9402ac): 4583 bytes on disk
I20240503 08:15:13.413357 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: UndoDeltaBlockGCOp(da529c8f8de241d38678d1596b9402ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20240503 08:15:13.495213 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:15:13.572609 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:13.624117 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling FlushMRSOp(585ba864cc384e7f9db90e9d4c47e82c): perf score=0.033397
W20240503 08:15:13.637950 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:13.647998 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling FlushMRSOp(564fb1b51abd4f8da0372ec99e3c599c): perf score=0.033352
I20240503 08:15:13.661921 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: FlushMRSOp(585ba864cc384e7f9db90e9d4c47e82c) complete. Timing: real 0.037s user 0.030s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":327,"dirs.run_cpu_time_us":304,"dirs.run_wall_time_us":2016,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":600,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":464,"threads_started":2,"wal-append.queue_time_us":333}
I20240503 08:15:13.663125 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling UndoDeltaBlockGCOp(585ba864cc384e7f9db90e9d4c47e82c): 4583 bytes on disk
I20240503 08:15:13.663795 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: UndoDeltaBlockGCOp(585ba864cc384e7f9db90e9d4c47e82c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240503 08:15:13.686766 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: FlushMRSOp(564fb1b51abd4f8da0372ec99e3c599c) complete. Timing: real 0.039s user 0.027s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":371,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1885,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":724,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":474,"threads_started":2,"wal-append.queue_time_us":320}
I20240503 08:15:13.687786 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling UndoDeltaBlockGCOp(564fb1b51abd4f8da0372ec99e3c599c): 4581 bytes on disk
I20240503 08:15:13.688295 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: UndoDeltaBlockGCOp(564fb1b51abd4f8da0372ec99e3c599c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240503 08:15:13.783118 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:15:14.056188 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:15:14.071810 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:14.164554 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:15:14.167670 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling FlushMRSOp(173376257c4946b5a801a27e4dffe90d): perf score=0.033402
I20240503 08:15:14.199836 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: FlushMRSOp(173376257c4946b5a801a27e4dffe90d) complete. Timing: real 0.032s user 0.017s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":383,"dirs.run_cpu_time_us":257,"dirs.run_wall_time_us":1818,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":656,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":520,"threads_started":2,"wal-append.queue_time_us":382}
I20240503 08:15:14.201086 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling UndoDeltaBlockGCOp(173376257c4946b5a801a27e4dffe90d): 4924 bytes on disk
I20240503 08:15:14.201709 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: UndoDeltaBlockGCOp(173376257c4946b5a801a27e4dffe90d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240503 08:15:14.217000 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:14.280058 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:15:14.418977 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling FlushMRSOp(4babf0ea9b954a23b1b1c4e403dce0bd): perf score=0.033381
I20240503 08:15:14.441809 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling FlushMRSOp(a20489bf65774902af38d228ee966ffe): perf score=0.033358
I20240503 08:15:14.456435 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: FlushMRSOp(4babf0ea9b954a23b1b1c4e403dce0bd) complete. Timing: real 0.037s user 0.027s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":362,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1686,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":565,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":462,"threads_started":2,"wal-append.queue_time_us":317}
I20240503 08:15:14.458076 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(4babf0ea9b954a23b1b1c4e403dce0bd): 4923 bytes on disk
I20240503 08:15:14.458669 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(4babf0ea9b954a23b1b1c4e403dce0bd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240503 08:15:14.475219 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: FlushMRSOp(a20489bf65774902af38d228ee966ffe) complete. Timing: real 0.033s user 0.022s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":317,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":1722,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":529,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":953,"threads_started":2,"wal-append.queue_time_us":841}
I20240503 08:15:14.476195 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling UndoDeltaBlockGCOp(a20489bf65774902af38d228ee966ffe): 4838 bytes on disk
I20240503 08:15:14.476850 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: UndoDeltaBlockGCOp(a20489bf65774902af38d228ee966ffe) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240503 08:15:14.637320 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:14.704465 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling FlushMRSOp(e9997e8da99c4f45a63bc3db521b33ba): perf score=0.033358
W20240503 08:15:14.706604 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:14.711014 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling FlushMRSOp(0caf86850b3841ea811b0a6ab19ae330): perf score=0.033343
I20240503 08:15:14.736689 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: FlushMRSOp(e9997e8da99c4f45a63bc3db521b33ba) complete. Timing: real 0.032s user 0.026s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":351,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1721,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":620,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":479,"threads_started":2,"wal-append.queue_time_us":341}
I20240503 08:15:14.737957 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling UndoDeltaBlockGCOp(e9997e8da99c4f45a63bc3db521b33ba): 4923 bytes on disk
I20240503 08:15:14.738444 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: UndoDeltaBlockGCOp(e9997e8da99c4f45a63bc3db521b33ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240503 08:15:14.744239 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: FlushMRSOp(0caf86850b3841ea811b0a6ab19ae330) complete. Timing: real 0.033s user 0.023s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":1460,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":736,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":4864,"thread_start_us":199,"threads_started":1,"wal-append.queue_time_us":331}
I20240503 08:15:14.745515 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(0caf86850b3841ea811b0a6ab19ae330): 4583 bytes on disk
I20240503 08:15:14.746040 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(0caf86850b3841ea811b0a6ab19ae330) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20240503 08:15:15.107774 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:15:15.170002 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling FlushMRSOp(b4ceeeb3495e4a8fac6e13e431c64af4): perf score=0.033362
I20240503 08:15:15.200990 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: FlushMRSOp(b4ceeeb3495e4a8fac6e13e431c64af4) complete. Timing: real 0.031s user 0.026s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":350,"dirs.run_cpu_time_us":288,"dirs.run_wall_time_us":1744,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":706,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":438,"threads_started":2,"wal-append.queue_time_us":299}
I20240503 08:15:15.202088 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling UndoDeltaBlockGCOp(b4ceeeb3495e4a8fac6e13e431c64af4): 4923 bytes on disk
I20240503 08:15:15.202623 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: UndoDeltaBlockGCOp(b4ceeeb3495e4a8fac6e13e431c64af4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
W20240503 08:15:15.206501 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:15:15.207381 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:15.244429 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:15:15.321149 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:15.375509 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:15:15.474821 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 08:15:15.475765 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:15:15.637521 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:15.706785 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:15.732187 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling FlushMRSOp(ac392ff7965a405f96efc777aca4f0bb): perf score=0.033368
I20240503 08:15:15.776465 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: FlushMRSOp(ac392ff7965a405f96efc777aca4f0bb) complete. Timing: real 0.044s user 0.031s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":457,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":1806,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":763,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":498,"threads_started":2,"wal-append.queue_time_us":317}
I20240503 08:15:15.777575 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling UndoDeltaBlockGCOp(ac392ff7965a405f96efc777aca4f0bb): 4838 bytes on disk
I20240503 08:15:15.778192 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: UndoDeltaBlockGCOp(ac392ff7965a405f96efc777aca4f0bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240503 08:15:15.804546 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:15:15.818641 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:15:15.956023 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling FlushMRSOp(a20489bf65774902af38d228ee966ffe): perf score=0.033370
W20240503 08:15:15.988195 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:15:16.010743 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: FlushMRSOp(a20489bf65774902af38d228ee966ffe) complete. Timing: real 0.054s user 0.045s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":310,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1730,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":726,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":417,"threads_started":2,"wal-append.queue_time_us":302}
I20240503 08:15:16.011761 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling UndoDeltaBlockGCOp(a20489bf65774902af38d228ee966ffe): 4838 bytes on disk
I20240503 08:15:16.012262 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: UndoDeltaBlockGCOp(a20489bf65774902af38d228ee966ffe) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240503 08:15:16.212451 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:16.279151 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:16.331457 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:15:16.550182 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:15:16.553673 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:15:16.702196 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:16.825449 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:17.257599 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:15:17.326130 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:17.376127 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:17.585873 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:15:17.621274 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:15:17.758705 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling FlushMRSOp(c6e1b9f104ea471cab7527bd56433327): perf score=0.033383
W20240503 08:15:17.766623 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:17.791509 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: FlushMRSOp(c6e1b9f104ea471cab7527bd56433327) complete. Timing: real 0.033s user 0.026s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":379,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1047,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":582,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":434,"threads_started":2,"wal-append.queue_time_us":276}
I20240503 08:15:17.792647 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(c6e1b9f104ea471cab7527bd56433327): 4921 bytes on disk
I20240503 08:15:17.793195 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(c6e1b9f104ea471cab7527bd56433327) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240503 08:15:17.834169 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:17.840729 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:15:18.091370 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:15:18.156119 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:15:18.237717 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:18.251461 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling FlushMRSOp(d13c33b8712e4c488abb99b73ff42b86): perf score=0.033387
W20240503 08:15:18.261171 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:18.261618 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:15:18.288107 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: FlushMRSOp(d13c33b8712e4c488abb99b73ff42b86) complete. Timing: real 0.036s user 0.020s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":442,"dirs.run_cpu_time_us":271,"dirs.run_wall_time_us":2058,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":583,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":515,"threads_started":2,"wal-append.queue_time_us":287}
I20240503 08:15:18.289251 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling UndoDeltaBlockGCOp(d13c33b8712e4c488abb99b73ff42b86): 4583 bytes on disk
I20240503 08:15:18.289749 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: UndoDeltaBlockGCOp(d13c33b8712e4c488abb99b73ff42b86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240503 08:15:18.290565 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling FlushMRSOp(b5736ee8065344ea8124fdf759035216): perf score=0.033357
I20240503 08:15:18.345415 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: FlushMRSOp(b5736ee8065344ea8124fdf759035216) complete. Timing: real 0.055s user 0.030s sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1600,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":621,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":220,"threads_started":1,"wal-append.queue_time_us":320}
I20240503 08:15:18.346388 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling UndoDeltaBlockGCOp(b5736ee8065344ea8124fdf759035216): 4838 bytes on disk
I20240503 08:15:18.346853 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: UndoDeltaBlockGCOp(b5736ee8065344ea8124fdf759035216) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240503 08:15:18.380033 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:15:18.635197 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:18.790141 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:18.995682 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:15:19.022145 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:15:19.022287 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling FlushMRSOp(2d113e8de02f447b9da0a1cb03686cd2): perf score=0.033352
W20240503 08:15:19.022572 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:15:19.055163 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: FlushMRSOp(2d113e8de02f447b9da0a1cb03686cd2) complete. Timing: real 0.033s user 0.013s sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":331,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1528,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":634,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":420,"threads_started":2,"wal-append.queue_time_us":309}
I20240503 08:15:19.056207 30722 maintenance_manager.cc:419] P b29361eb4afa4ab3af7d8019d8a5042d: Scheduling UndoDeltaBlockGCOp(2d113e8de02f447b9da0a1cb03686cd2): 4838 bytes on disk
I20240503 08:15:19.056746 30613 maintenance_manager.cc:643] P b29361eb4afa4ab3af7d8019d8a5042d: UndoDeltaBlockGCOp(2d113e8de02f447b9da0a1cb03686cd2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240503 08:15:19.103695 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:15:19.274534 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:19.301365 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:15:19.528134 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:19.655283 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:19.887399 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:15:19.915724 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:20.054525 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:20.255149 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:15:20.333500 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:15:20.334455 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:15:20.415338 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:20.483749 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:15:20.484922 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:15:20.593225 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:15:20.648583 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:20.795742 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:15:20.936261 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:15:20.998829 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:21.070536 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:21.455667 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:21.488762 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:21.547627 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:15:21.608672 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:15:21.742020 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:15:21.867769 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:21.942474 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:22.451733 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:22.485518 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:15:22.533558 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:22.696943 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:15:22.916855 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:22.976068 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:15:23.043294 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:15:23.062795 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:15:23.106320 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:23.272183 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:15:23.312688 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:23.384399 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:15:23.521744 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:23.570606 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:15:23.769192 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:23.913538 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:15:23.988349 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:15:24.081945 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:24.111117 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:15:24.274871 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:24.281412 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:15:24.566501 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:24.715039 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:24.973977 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:25.265180 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:25.331626 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:15:25.373764 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:15:25.404610 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 08:15:25.405683 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:25.467298 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:15:25.471823 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:15:25.562326 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:15:25.719053 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:25.785619 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:15:25.890323 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:15:25.950842 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:26.009881 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:15:26.152411 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:15:26.166908 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:26.482532 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:26.482623 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:15:26.512995 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:15:26.693668 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:26.952339 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:27.286088 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:27.523425 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:27.703452 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:15:27.730226 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:15:27.731815 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:15:27.784606 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:15:27.817934 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:27.939821 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:27.995970 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:15:28.335506 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:28.382189 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:28.508985 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:15:28.534073 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:15:28.662254 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:15:28.691040 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:15:28.813480 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:15:28.826495 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:28.913602 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:29.147812 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:15:29.275347 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:29.440991 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:29.659950 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:15:29.764186 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:29.900174 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:30.053346 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:15:30.130106 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:15:30.160773 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:15:30.197458 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:30.202126 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:15:30.327324 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:30.511719 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:15:30.715513 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:15:30.722779 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:30.812491 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:30.962611 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:15:31.090554 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:15:31.234730 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:31.316453 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:15:31.402424 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:31.448369 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:15:31.464701 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:15:31.782872 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:31.874610 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:15:31.965759 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:32.184464 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:32.494964 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:32.563649 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:15:32.569403 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:15:32.664172 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:32.665102 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:15:32.857975 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:15:32.858883 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:15:32.971709 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:33.213021 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:33.415370 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:33.539343 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:15:33.747882 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:33.790771 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:15:33.883744 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:15:33.914651 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:15:33.951608 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:15:34.015703 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:34.185165 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:34.430835 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:15:34.618379 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:34.717895 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:34.856073 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:15:34.857014 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:15:35.005656 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:15:35.055626 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:35.128508 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:35.162469 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:15:35.240067 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:15:35.537644 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 36 similar messages]
W20240503 08:15:35.538720 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:35.620118 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:36.027321 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:15:36.028203 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:15:36.078462 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:36.078575 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:36.109669 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:15:36.326967 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:15:36.586592 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:36.589223 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:15:36.627602 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:36.631973 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:15:37.184489 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:37.214417 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:15:37.245853 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:37.339810 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:15:37.471907 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:15:37.687093 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:15:37.774072 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:37.821027 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:37.870870 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:15:37.982844 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:15:38.147068 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:15:38.182600 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:38.343094 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:38.743045 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:15:38.805899 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:38.858945 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:38.882726 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:15:39.015892 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:15:39.298310 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:15:39.410197 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:39.426925 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:39.651540 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:15:39.667630 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:15:39.797459 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:15:39.866235 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:39.867843 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:39.942520 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:15:40.079456 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:15:40.134196 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:15:40.268378 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:40.344566 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:15:40.449398 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:40.730046 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:15:40.730945 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:41.000411 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:41.185513 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:15:41.186524 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:15:41.207690 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:41.386718 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:15:41.398975 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:41.677621 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:15:41.763953 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:15:41.783600 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:41.985668 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:41.991237 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:15:42.170047 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:42.199352 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:15:42.332723 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:15:42.472941 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:42.586930 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:15:42.655582 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:15:42.738826 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:42.820766 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:15:43.031132 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:43.115418 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 08:15:43.169432 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:43.567641 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:43.612040 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:43.635419 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:15:43.923622 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:15:44.026096 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:15:44.027643 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:44.135134 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:44.427934 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:15:44.521536 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:44.560887 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:44.727383 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:15:44.737560 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:15:44.774984 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:15:45.028991 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:15:45.099696 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:45.137001 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:45.212579 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:15:45.286626 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:15:45.384431 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:15:45.508927 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:45.704186 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:45.843735 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:15:45.933642 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:45.982725 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:15:46.103375 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:46.215526 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 08:15:46.291204 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:15:46.339866 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:46.670475 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:15:46.679423 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:46.954879 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:47.037029 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:15:47.101960 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:15:47.183933 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:15:47.238821 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:15:47.286650 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:47.380436 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:47.611824 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:15:47.642822 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:15:47.725661 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:15:47.762632 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:47.817968 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:48.163024 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:15:48.280136 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:48.287860 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:48.439781 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:15:48.809525 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:48.820384 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:48.833564 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:15:49.087181 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:15:49.326683 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:49.388456 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:49.458308 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:15:49.634259 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:15:49.699388 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:15:49.817405 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:49.856441 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:15:49.886265 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:50.178828 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:15:50.242496 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:15:50.273316 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:15:50.308916 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:15:50.392833 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:50.409034 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:50.851148 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:15:50.925133 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:15:50.926064 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:50.977974 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:51.188561 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:15:51.351938 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:15:51.380425 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:15:51.395117 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:51.442770 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:51.919406 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:15:51.991814 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:52.014993 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:52.209187 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:15:52.240687 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:15:52.324746 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:15:52.434355 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:52.440006 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:15:52.602536 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:52.681489 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:15:52.828464 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:15:52.864291 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:53.171074 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:15:53.226239 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:53.292153 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:53.537292 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:15:53.614267 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:53.815476 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:53.884936 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:15:53.932499 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:15:54.222419 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:54.391042 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:54.539948 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:15:54.705610 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:15:54.797534 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:54.804759 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:54.991178 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:15:54.994580 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:15:55.083701 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:15:55.240731 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:15:55.253098 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:15:55.307936 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:55.403024 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:55.640461 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:15:55.907058 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:55.931433 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:15:55.951089 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:15:55.952028 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:56.395982 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:56.404212 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:56.406064 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:15:56.474485 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:15:56.486748 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:15:56.819517 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:56.932252 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:57.284960 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:15:57.301944 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:57.364912 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:15:57.451831 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:15:57.507684 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:57.518206 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:15:57.573005 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:15:57.580543 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:15:57.686620 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:57.761479 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:15:58.073228 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:58.147481 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:58.546149 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:15:58.574026 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:58.579926 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:15:58.634182 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:58.773660 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:15:58.974983 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:59.213399 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:15:59.226451 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:59.481843 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:59.606045 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:15:59.807260 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:15:59.820565 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:15:59.936128 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:16:00.010118 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:00.015475 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:16:00.018585 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:16:00.046912 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:16:00.267925 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:00.289727 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:16:00.502465 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:00.691728 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:00.954058 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:16:00.955281 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:01.038756 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:16:01.088233 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:16:01.118786 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:16:01.178781 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:01.450707 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:01.550617 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:16:01.572973 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:01.654250 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:16:01.971369 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:02.088397 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:02.188417 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:16:02.300175 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:16:02.321045 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:16:02.361142 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:16:02.553586 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:02.638211 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:02.648231 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:16:02.714020 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:16:02.951447 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:16:03.030956 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:03.094321 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:03.556422 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:03.584798 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:16:03.601941 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:16:03.640635 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:16:03.648151 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:04.002602 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:04.162048 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:04.221324 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:16:04.621973 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:04.631155 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:16:04.745626 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:04.769673 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:16:04.895504 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:16:05.010673 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:05.010834 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:16:05.149755 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:05.273319 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:16:05.287631 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:16:05.427783 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:16:05.433051 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:05.669245 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:05.911857 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:06.121615 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:16:06.122468 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:16:06.134469 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:06.287487 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:06.372565 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:16:06.473657 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:16:06.612435 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:16:06.747113 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:06.785362 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:06.798583 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:16:07.150504 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:07.207376 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:07.254177 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:16:07.338536 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:16:07.385797 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:16:07.560369 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:16:07.726620 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:07.734465 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:16:07.816264 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:07.867717 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:16:07.906793 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:16:08.340348 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:16:08.342767 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:08.351135 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:08.745285 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:08.797657 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:16:08.832577 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:08.882123 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:16:09.186781 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:09.349483 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:09.398200 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:16:09.594640 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:09.861285 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:16:09.958811 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:10.020292 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:10.041229 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:16:10.092823 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:16:10.139001 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:16:10.258599 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:16:10.353771 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:16:10.543934 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:10.545990 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:16:10.592435 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:10.726689 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:16:11.090271 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:11.094133 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:11.193926 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:16:11.253633 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:16:11.541802 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:16:11.613157 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:16:11.630699 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:11.698621 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:12.113157 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:12.176898 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:12.195181 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:16:12.394371 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:16:12.585584 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:12.609314 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:16:12.621995 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:16:12.673897 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:12.681249 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:16:12.731165 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:16:12.957429 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:16:13.107656 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:16:13.170991 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:13.208029 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:13.423872 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:16:13.586297 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:16:13.607143 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:13.666105 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:13.915602 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:16:14.068280 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:14.084786 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:14.478150 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:14.655534 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:14.698760 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:16:14.800187 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:16:14.851903 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:16:14.933221 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:16:14.962554 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:15.003003 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:16:15.250429 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:15.491403 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:16:15.515698 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:15.712297 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:15.744958 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:16:15.794407 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:16:15.936167 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:15.939209 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:16:16.066134 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:16:16.241565 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:16:16.242499 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:16.522413 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:16.621506 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:16:16.687801 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:16.862416 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:16:17.067205 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:16:17.081024 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:17.180972 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:16:17.265517 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:17.326400 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:16:17.334705 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:16:17.477814 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:17.651655 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:16:17.808729 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:17.904178 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:17.937187 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:16:18.267318 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:16:18.269284 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:16:18.360965 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:16:18.390916 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:18.445015 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:18.756956 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:16:18.940459 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:19.066903 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:19.081441 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:16:19.312654 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:16:19.489109 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:19.595654 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:19.660480 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:16:19.660754 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:16:19.916224 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:16:19.928398 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:20.029559 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:20.119441 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:16:20.466681 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:20.513178 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:16:20.587040 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:16:20.625267 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:16:20.639492 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:20.903616 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:16:21.005347 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:21.154368 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:21.170944 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:16:21.522356 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:16:21.523226 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:21.671892 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:16:21.776198 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:21.902283 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:16:21.917917 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:21.957500 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:16:21.978013 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:16:22.142717 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:16:22.168375 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:22.454241 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:22.506265 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:16:22.691707 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:16:22.747583 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:23.042712 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:23.148811 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:16:23.152431 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:23.179572 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:16:23.249922 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:16:23.395830 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:16:23.436822 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:23.441263 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:16:23.718676 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:23.990252 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:24.242998 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:24.305617 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:16:24.306614 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:16:24.360658 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:16:24.455587 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:24.605015 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:16:24.772217 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:24.973954 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:16:25.021874 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:25.289211 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:16:25.389170 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:25.617090 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:25.658263 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:16:25.711791 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:16:25.769361 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:16:25.778720 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:16:25.944622 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:26.020207 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:26.072417 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:16:26.413046 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:26.587260 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:16:26.588105 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:26.718917 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:16:26.812141 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:16:26.813182 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:16:26.850556 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:16:26.933341 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:27.014348 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:27.216799 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:16:27.409566 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:16:27.457696 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:27.631593 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:27.914633 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:16:27.917466 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:16:27.995064 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:28.033659 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:28.137104 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:16:28.158999 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:16:28.359467 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:16:28.378055 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:28.599720 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:28.723804 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:16:28.972949 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:29.073897 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:29.224319 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:16:29.261466 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:16:29.396085 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:29.405686 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:16:29.407060 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:16:29.610993 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:29.761430 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:16:30.012517 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:30.211484 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:30.271385 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:16:30.339663 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:16:30.545351 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:16:30.570258 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:30.679026 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:16:30.834386 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:30.881960 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:16:31.166292 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:31.334410 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:31.371541 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:16:31.567838 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:31.775380 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:16:31.776500 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:31.843778 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:16:31.880306 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:16:31.880954 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:16:31.893283 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:16:32.063867 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:32.305056 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:32.370491 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:16:32.542399 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:32.644623 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:16:32.756490 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:32.811168 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:16:32.952594 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:16:33.085116 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:33.140961 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:16:33.160360 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:33.233556 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:16:33.419405 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:16:33.597326 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:33.678040 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:34.153070 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:34.196256 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:16:34.204667 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:34.267335 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:16:34.273361 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:16:34.555589 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:16:34.556565 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:16:34.656281 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:34.666214 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:34.810963 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:16:35.144601 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:16:35.151269 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:35.174855 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:35.295465 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:16:35.464841 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:16:35.584848 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:35.658980 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:35.677884 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:16:35.941731 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:16:36.058632 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:36.195246 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:36.528185 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:36.684713 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:16:36.783732 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:16:36.784770 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:16:36.813851 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:36.895012 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:16:36.903548 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:16:36.954550 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:37.109303 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:16:37.299242 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:37.445940 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:16:37.532529 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:37.592087 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:16:37.647017 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:16:37.859964 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:37.979002 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:16:38.077535 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:16:38.131814 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:38.207549 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:16:38.279204 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:38.596307 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:16:38.679862 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:38.766407 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:38.982021 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:16:39.252944 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:39.270988 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:39.303995 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:16:39.319557 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:16:39.426370 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:16:39.712716 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:16:39.809988 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:39.821856 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:39.887295 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:16:39.901743 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:16:40.198890 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:40.236009 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:16:40.390187 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:40.430059 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:16:40.601586 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:16:40.677397 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:40.814620 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:40.965113 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:16:41.086187 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:41.436654 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:41.484380 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:16:41.569609 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:16:41.650295 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:41.858860 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:16:41.866684 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:16:41.944115 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:16:42.018672 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:42.053164 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:42.069864 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:16:42.316854 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:16:42.348778 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:16:42.544031 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:42.586320 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:16:42.615500 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:42.730165 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:16:42.920106 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:43.133440 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:43.317625 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:16:43.318542 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:16:43.367031 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:43.391829 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:16:43.558905 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:43.781744 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:43.972568 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:44.103209 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:16:44.178365 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:16:44.248236 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:16:44.268182 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:44.374748 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:44.641916 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:16:44.674695 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:16:44.711773 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:44.745847 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:16:44.773772 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:45.164600 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:16:45.230826 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:16:45.235281 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:16:45.317122 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:45.363595 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:45.698275 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:45.891948 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:46.029213 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:16:46.082484 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:46.105501 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:16:46.288630 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:46.447115 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:16:46.512851 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:16:46.651293 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:46.748930 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:16:46.822207 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:47.032415 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:16:47.037290 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:16:47.074174 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:47.085811 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:16:47.086709 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:16:47.210640 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:16:47.214053 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:47.468675 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:16:47.513492 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:16:47.613112 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:47.654090 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:48.157487 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:48.223682 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:48.565511 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:16:48.566515 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:16:48.596338 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:48.807415 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:16:48.841595 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:48.930164 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:16:48.965925 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:16:49.004032 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:49.262980 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:49.372201 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:16:49.383172 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:16:49.458190 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:49.614017 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:16:49.739373 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:16:49.814968 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:49.822520 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:16:50.031993 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:16:50.063401 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:50.268044 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:50.460558 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:16:50.556074 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:50.778569 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:51.042833 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:51.227808 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:16:51.229718 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:51.261773 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:16:51.283818 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:16:51.523743 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:16:51.525249 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:16:51.606429 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:51.842650 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:51.952582 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:16:52.033571 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:16:52.065702 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:16:52.066769 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:52.120645 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:16:52.127489 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:16:52.277897 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:52.381295 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:16:52.604489 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:16:52.605441 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:52.657047 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:53.130808 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:53.209889 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:53.582118 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:53.625841 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:53.710086 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:16:53.711685 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:16:53.751684 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:16:53.787036 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:16:54.013293 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:16:54.112636 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:16:54.185007 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:54.187965 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:54.365260 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:16:54.642750 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:16:54.644358 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:54.687182 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:16:54.710175 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:54.760648 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:16:55.048223 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:55.198269 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:55.425505 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:16:55.571964 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:55.597232 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:55.842741 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:16:55.947816 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:56.141098 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:56.209175 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:16:56.295619 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:16:56.527878 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:16:56.533653 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:56.564988 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:16:56.647279 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:16:56.752557 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:56.843225 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:16:56.998108 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:57.125197 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:16:57.125999 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:16:57.190872 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:16:57.191695 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:57.355046 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:16:57.405570 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:57.475003 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:16:57.696167 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:57.780849 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:57.958199 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:16:58.234505 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:58.388952 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:58.681505 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:58.774571 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:58.945466 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:16:58.946525 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:16:58.979713 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:16:59.001699 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:16:59.157645 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:16:59.234804 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:59.272858 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:59.390269 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:16:59.398933 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:16:59.646044 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:16:59.680094 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:16:59.793555 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:16:59.875061 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:00.243939 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:17:00.277840 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:00.363942 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:00.469497 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:17:00.803160 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:00.821079 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:00.947918 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:17:01.281268 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:17:01.338719 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:01.421430 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:01.569875 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:17:01.582954 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:17:01.753136 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:17:01.760627 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:17:01.895589 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:01.965792 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:02.077303 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:17:02.204968 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:17:02.302093 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:17:02.315835 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:17:02.404336 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:02.502722 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:02.569604 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:17:02.819092 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:17:02.888247 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:02.921128 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:03.163151 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:17:03.386835 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:03.462013 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:03.888928 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:03.905740 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:03.931910 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:17:03.989094 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:17:04.076040 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:17:04.306308 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:17:04.413378 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:17:04.414522 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:17:04.481351 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:04.505244 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:04.633256 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:17:04.677507 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:17:04.988422 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:05.017586 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:05.078790 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:17:05.335582 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:17:05.483199 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:05.587706 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:05.707113 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:17:06.007354 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:06.041385 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:17:06.176673 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:06.290598 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:17:06.423192 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:06.479729 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:17:06.498944 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:17:06.581993 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:06.755903 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:17:06.831914 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:06.967561 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:17:07.098408 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:17:07.174666 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:07.223244 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:17:07.395869 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:17:07.396988 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:07.410842 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:17:07.653635 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:17:07.753937 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:07.897469 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:07.965117 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:17:08.015138 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:17:08.363018 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:08.411499 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:08.457360 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:17:08.795099 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:08.907868 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:09.120854 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:17:09.184736 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:09.186480 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:17:09.323453 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:09.440950 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:17:09.441991 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:17:09.487485 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:17:09.493912 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:17:09.654917 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:09.699563 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:17:09.732268 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:09.953181 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:17:10.245741 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:17:10.275130 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:10.278391 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:10.324393 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:17:10.651727 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:10.710255 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:17:10.753062 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:11.168376 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:17:11.179873 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:11.290635 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:11.681975 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:17:11.729893 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:11.795753 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:17:11.875792 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:11.894665 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:17:11.929344 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:17:12.000154 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:17:12.156028 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:17:12.198848 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:17:12.308660 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:12.409379 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:17:12.420444 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:12.422350 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:17:12.792919 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:12.934196 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:17:12.937294 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:17:13.026741 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:13.111196 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:17:13.233978 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:13.460446 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:13.767881 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:13.856738 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:14.028613 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:17:14.144937 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:17:14.234719 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:14.281085 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:14.382462 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:17:14.477188 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:17:14.521667 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:17:14.604609 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:17:14.760264 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:14.808241 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:17:14.809456 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:17:14.811030 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:15.220852 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:15.357645 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:15.388407 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:17:15.495972 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:17:15.573789 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:17:15.743675 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:15.794947 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:16.186952 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:16.201150 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:16.290791 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:17:16.333112 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:17:16.702004 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:17:16.734023 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:16.743631 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:16.898454 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:17:17.007059 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:17:17.135113 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:17.158998 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:17.224797 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:17:17.240383 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:17:17.421254 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:17:17.553354 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:17:17.616151 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 08:17:17.617324 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:17.721395 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:17:17.725140 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:18.005570 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:17:18.135581 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:17:18.182981 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:18.336719 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:18.656700 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:18.754281 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:18.937884 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:17:19.055341 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:17:19.139232 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:19.313831 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:19.402420 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:17:19.424444 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:17:19.554423 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:17:19.597798 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:19.669494 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:17:19.718715 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:19.779147 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:17:20.008674 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:17:20.144440 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:17:20.154527 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:20.238116 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:20.430437 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:17:20.536485 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:17:20.732133 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:20.759148 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:21.171012 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:21.305625 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:21.360189 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:17:21.526685 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:17:21.585526 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:17:21.721733 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:21.766199 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:21.918370 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:17:21.952343 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:17:22.071980 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:17:22.154877 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:17:22.173785 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:17:22.318426 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:22.346076 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:22.385399 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:17:22.587328 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:17:22.703470 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:17:22.704363 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:22.721936 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:22.814991 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:17:22.819864 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:17:23.111943 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:23.197407 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:23.688946 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:23.742552 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:23.915669 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:17:24.082115 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:17:24.121997 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:24.173992 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:17:24.254814 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:17:24.299248 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:24.371052 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:17:24.485143 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:17:24.559877 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:17:24.658905 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:24.676129 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:24.916427 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:17:25.223578 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:25.244181 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:25.384013 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:17:25.401201 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:17:25.619279 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:17:25.668534 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:25.837314 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:26.062295 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:26.267864 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:26.337357 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:17:26.445842 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:26.604418 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:17:26.712194 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:17:26.755939 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:17:26.755920 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:17:26.756667 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:17:26.810312 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:26.904748 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:27.073801 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:17:27.100601 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:17:27.248739 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:27.279984 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:27.316174 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:17:27.626731 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:17:27.710711 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:17:27.711532 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:27.767521 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:27.866369 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:17:28.099822 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:28.189550 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:28.242177 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:17:28.618083 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:28.728435 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:28.938235 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:17:29.073179 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:29.121065 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:29.213634 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:17:29.322605 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:17:29.417928 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:17:29.448024 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:29.502580 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:17:29.545163 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:17:29.644155 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:29.841220 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:29.882441 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:17:29.909668 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:17:30.119191 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:30.374130 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:17:30.429765 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:30.731588 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:30.837667 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:17:31.018529 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:17:31.019778 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:31.290393 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:31.295714 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:17:31.424499 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:31.709380 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:17:31.753830 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:31.785873 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:17:31.922468 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:17:31.924048 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:31.928479 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:17:31.950178 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:17:32.002048 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:17:32.161422 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:32.168155 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:17:32.355268 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:17:32.357825 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:32.696789 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:32.733139 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:17:32.757200 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:17:32.773039 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:32.983670 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:17:33.212914 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:33.278659 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:33.553969 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:17:33.717137 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:33.761261 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:33.960096 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:17:33.991401 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:17:34.191164 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:17:34.201926 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:34.243481 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:34.390645 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:17:34.488726 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:17:34.491255 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:17:34.656077 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:17:34.720310 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:34.730422 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:34.843546 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:17:35.166229 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:17:35.189647 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:35.237018 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:35.759618 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:35.764225 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:36.347555 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:36.381094 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:17:36.382015 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:36.387661 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:17:36.474015 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:17:36.654843 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:17:36.695241 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:17:36.818827 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:17:36.872174 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:36.909651 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:17:36.943289 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:37.171602 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:17:37.246028 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:17:37.247071 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:17:37.310169 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:17:37.420694 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:37.552414 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:37.756165 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:17:37.815707 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:17:37.855785 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:17:37.909031 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:38.047739 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:38.328243 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:38.429637 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:38.797484 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:17:38.808344 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:38.832129 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:39.202651 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:17:39.245903 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:17:39.299037 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:39.419972 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:39.429203 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:17:39.460419 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:17:39.732971 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:39.784291 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:17:39.790513 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:17:39.830528 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:17:39.865049 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:17:40.040799 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:40.260412 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:40.446218 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:17:40.531737 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:40.814329 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:41.042644 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:41.215447 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:17:41.227488 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:41.541090 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:41.607888 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:17:41.614847 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:41.876649 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:17:41.945755 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:41.948771 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:17:41.962116 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:17:41.979974 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:17:42.023259 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:17:42.042268 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:17:42.185441 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:42.327118 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:42.438639 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:17:42.529174 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:17:42.552855 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:17:42.731564 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:42.931692 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:17:42.932529 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:42.934259 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:17:43.046056 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:17:43.293244 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:43.467312 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:43.786798 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:17:43.853613 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:43.883401 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:44.239836 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:17:44.316444 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:44.322294 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:44.437294 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:17:44.442469 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:17:44.464457 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:17:44.584790 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:17:44.796257 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:44.893826 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:17:44.897401 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:45.257309 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:17:45.357936 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:17:45.413597 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:45.439199 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:45.595706 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:17:45.993081 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:46.053254 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:46.320490 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:17:46.479960 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:46.610074 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:17:46.646224 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:46.699734 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:17:46.837775 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:17:46.896103 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:17:46.993325 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:17:46.995576 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240503 08:17:47.042776 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:47.088675 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:17:47.193689 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:47.331665 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:17:47.578584 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:47.621094 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:47.805145 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:17:47.951417 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:17:48.026274 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:17:48.054096 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:17:48.075310 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:17:48.076136 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:48.085124 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:48.595081 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:48.685801 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:48.779973 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:17:49.118505 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:49.213625 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:17:49.273173 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:49.299273 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:17:49.312098 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:17:49.319419 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:17:49.630113 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:17:49.720727 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:49.807250 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:49.967296 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:17:50.183676 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:50.229168 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:50.321727 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:17:50.456561 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:17:50.628731 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:50.652132 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:17:50.835021 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:51.250952 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:51.304859 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:51.313561 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:17:51.643510 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:17:51.754827 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:51.829066 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:17:51.875267 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:17:51.929131 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:51.971834 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:17:52.202761 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:17:52.223768 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:52.293706 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:17:52.353894 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:17:52.414763 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:17:52.528725 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:52.712683 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:52.905057 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:17:52.908910 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:17:52.951627 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:53.059387 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:17:53.318749 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:17:53.328433 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:17:53.331753 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:53.358078 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:53.707955 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:53.833983 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:17:53.855681 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:54.169438 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:17:54.212792 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:17:54.247939 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:17:54.314774 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:54.450268 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:17:54.457698 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:54.839063 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:17:54.859841 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:55.048277 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:55.117947 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:17:55.118041 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:17:55.280201 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:55.458058 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:55.466820 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:17:55.730300 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:55.806983 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:17:55.991691 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:56.035663 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:17:56.217983 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:56.606979 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:56.738762 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:56.786031 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:17:56.849671 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:17:56.865938 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:17:57.037765 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:17:57.134438 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:57.143795 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:17:57.200891 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:57.440739 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:17:57.503218 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:17:57.599030 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:57.604893 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:57.615357 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:17:57.745378 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:17:58.115821 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:58.150171 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:17:58.164950 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:17:58.181277 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:58.510694 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:17:58.545842 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:17:58.561898 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:17:58.610692 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:58.708333 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:59.004760 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:59.121878 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:17:59.207585 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:59.286212 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:17:59.516997 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:59.538977 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:17:59.623658 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:17:59.663671 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:17:59.815155 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:17:59.948897 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:00.043459 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:18:00.212018 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:00.291313 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:18:00.445065 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:00.619669 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:18:00.728111 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:00.749966 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:18:00.931569 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:18:01.069267 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:01.267416 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:01.572856 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:01.611618 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:18:01.633996 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:18:01.771479 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:01.799353 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:18:02.022696 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:02.162482 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:02.236307 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:18:02.279067 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:18:02.329280 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:18:02.455469 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:02.615157 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:18:02.636132 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:18:02.676297 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:02.715607 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:18:02.948673 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:03.052745 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:03.161330 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:18:03.175014 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:18:03.345942 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:18:03.478217 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:03.529093 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 08:18:03.529986 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:03.559823 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:18:03.998442 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:04.033437 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:04.045660 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:18:04.061900 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:18:04.156301 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:18:04.375614 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:04.577207 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:04.762755 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:18:04.879215 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:04.927531 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:18:04.990608 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:04.992209 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:18:05.063264 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:18:05.304385 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:05.515431 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:18:05.540364 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:05.680402 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:05.747473 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:18:06.020015 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:06.135800 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:06.150131 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:18:06.571980 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:18:06.603963 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:06.628152 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:18:06.695199 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:06.823791 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:18:06.920105 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:18:07.028980 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:07.168478 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:07.304231 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:18:07.511569 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:18:07.534680 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:07.557927 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:18:07.681511 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:18:07.684221 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:07.747337 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:18:07.950774 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:18:08.047446 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:08.155855 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:18:08.287766 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:08.381072 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:18:08.634629 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:18:08.635597 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:18:08.635849 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:08.682876 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:08.923099 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:18:09.107548 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:09.120249 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:18:09.137091 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:09.198696 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:18:09.219002 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:18:09.542902 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:09.636410 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:09.752177 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:18:09.837468 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:18:09.868470 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:18:09.942473 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:10.114001 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:10.409678 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:10.474421 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:18:10.540361 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:10.723738 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:18:10.895146 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:18:10.900560 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:11.151654 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:11.333848 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:18:11.404227 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:11.662919 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:18:11.674075 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:18:11.687719 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:18:11.689204 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:11.881193 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:18:11.896073 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:12.182572 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:12.191757 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:18:12.255401 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:18:12.445600 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:12.721161 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:12.893826 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:12.959241 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:18:13.099480 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:18:13.133095 28700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:18:13.134311 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:18:13.284567 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:13.460150 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:18:13.500604 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:13.571065 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:18:13.634739 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:18:13.648550 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:18:13.904595 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 08:18:13.905519 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:13.977459 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:14.291833 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:18:14.342175 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:18:14.366643 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:14.393064 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:18:14.403462 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:14.703308 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:18:14.780156 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:14.990056 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:18:14.990176 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:15.400955 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:15.523674 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:18:15.559629 28700 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:18:15.589706 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:15.932649 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:15.994220 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:16.027864 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:18:16.202934 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:18:16.286607 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:18:16.378293 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:16.460273 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:16.707527 28440 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:18:16.899021 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:16.943864 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:17.057739 28309 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:18:17.086758 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:18:17.176434 28309 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:18:17.389473 32363 consensus_queue.cc:579] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Leader has been unable to successfully communicate with peer bea2145b8e24436392b18ec3cfe90b93 for more than 300 seconds (300.083s)
W20240503 08:18:17.391527 28570 consensus_peers.cc:462] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:18:17.398658 28630 consensus_queue.cc:237] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:18:17.400956 28374 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:18:17.401000 30672 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:18:17.401492 901 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:18:17.401762 900 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:18:17.405205 32363 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:18:17.405956 30672 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:18:17.408618 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d60a162155504975a39c20eff88341e1 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:18:17.410490 28232 catalog_manager.cc:5484] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:18:17.412247 28374 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
W20240503 08:18:17.426831 28219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:18:17.428519 28630 consensus_queue.cc:237] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
W20240503 08:18:17.429816 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d60a162155504975a39c20eff88341e1: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:18:17.430895 30672 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 08:18:17.431425 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d60a162155504975a39c20eff88341e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:17.431484 901 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:18:17.432500 28374 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:18:17.433310 901 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:18:17.434389 900 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:18:17.435107 30672 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:18:17.440493 28226 catalog_manager.cc:5484] T d60a162155504975a39c20eff88341e1 P b29361eb4afa4ab3af7d8019d8a5042d reported cstate change: config changed from index 8 to 9, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:18:17.442509 28217 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d60a162155504975a39c20eff88341e1 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:18:17.443296 28374 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:18:17.476888 32363 consensus_queue.cc:579] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Leader has been unable to successfully communicate with peer bea2145b8e24436392b18ec3cfe90b93 for more than 300 seconds (300.349s)
W20240503 08:18:17.478956 28570 consensus_peers.cc:462] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:18:17.481626 28630 consensus_queue.cc:237] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:18:17.486411 30672 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:18:17.486565 28761 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:18:17.487478 32363 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:18:17.488091 900 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:18:17.490746 900 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:18:17.492573 30672 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:18:17.495574 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 09156806b0744752969b6a574e24b50a with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:18:17.495921 28761 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:18:17.497292 28232 catalog_manager.cc:5484] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:18:17.505300 28630 consensus_queue.cc:237] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
W20240503 08:18:17.507643 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): Couldn't send request to peer 716ffbb2a00b4f43b03e5cedcc9d302a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09156806b0744752969b6a574e24b50a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:17.507736 30672 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:18:17.508309 28761 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:18:17.508278 901 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:18:17.509136 901 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:18:17.510116 28570 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:18:17.511427 900 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:18:17.512496 28761 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:18:17.512995 30672 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:18:17.516158 28217 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 09156806b0744752969b6a574e24b50a with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:18:17.518492 28232 catalog_manager.cc:5484] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 8 to 9, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:18:17.551488 900 consensus_queue.cc:579] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Leader has been unable to successfully communicate with peer bea2145b8e24436392b18ec3cfe90b93 for more than 300 seconds (300.039s)
I20240503 08:18:17.555701 28630 consensus_queue.cc:237] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:18:17.558107 30672 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:18:17.558794 900 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 08:18:17.559283 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): Couldn't send request to peer 716ffbb2a00b4f43b03e5cedcc9d302a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:17.559677 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:17.561766 901 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:18:17.562523 30672 raft_consensus.cc:2932] T b4ceeeb3495e4a8fac6e13e431c64af4 P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:18:17.567667 28226 catalog_manager.cc:5484] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 9 to 10, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:18:17.569119 28217 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b4ceeeb3495e4a8fac6e13e431c64af4 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:18:17.626286 846 consensus_queue.cc:579] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Leader has been unable to successfully communicate with peer bea2145b8e24436392b18ec3cfe90b93 for more than 300 seconds (300.092s)
I20240503 08:18:17.637584 28374 consensus_queue.cc:237] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:18:17.641693 30414 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:18:17.642359 28502 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:18:17.643725 846 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:18:17.644201 978 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:18:17.648533 846 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:18:17.649334 28502 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:18:17.650133 30414 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:18:17.652776 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 173376257c4946b5a801a27e4dffe90d with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:18:17.654209 28228 catalog_manager.cc:5484] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 7 to 8, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:18:17.659762 28374 consensus_queue.cc:237] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:18:17.662056 28502 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:18:17.662051 30412 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:18:17.662645 846 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:18:17.663203 978 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:18:17.664911 846 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:18:17.665575 30412 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:18:17.668438 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 173376257c4946b5a801a27e4dffe90d with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 08:18:17.670212 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 173376257c4946b5a801a27e4dffe90d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:17.670262 28226 catalog_manager.cc:5484] T 173376257c4946b5a801a27e4dffe90d P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: config changed from index 8 to 9, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:18:17.670526 28502 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:18:17.735867 663 consensus_queue.cc:579] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Leader has been unable to successfully communicate with peer bea2145b8e24436392b18ec3cfe90b93 for more than 300 seconds (300.082s)
I20240503 08:18:17.745128 28761 consensus_queue.cc:237] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } }
I20240503 08:18:17.747275 28630 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:18:17.747345 30412 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:18:17.747949 984 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:18:17.748266 945 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:18:17.751313 984 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:18:17.752077 28630 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:18:17.751983 30412 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:18:17.756496 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0caf86850b3841ea811b0a6ab19ae330 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:18:17.757961 28226 catalog_manager.cc:5484] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:18:17.763924 28761 consensus_queue.cc:237] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:18:17.766700 30412 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:18:17.767095 28630 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:18:17.767297 991 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 08:18:17.767684 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0caf86850b3841ea811b0a6ab19ae330. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:17.768075 663 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:18:17.769866 991 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:18:17.770496 30412 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:18:17.775053 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0caf86850b3841ea811b0a6ab19ae330 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:18:17.776471 28228 catalog_manager.cc:5484] T 0caf86850b3841ea811b0a6ab19ae330 P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: config changed from index 12 to 13, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:18:17.775748 28630 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:18:17.867480 846 consensus_queue.cc:579] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Leader has been unable to successfully communicate with peer bea2145b8e24436392b18ec3cfe90b93 for more than 300 seconds (300.302s)
I20240503 08:18:17.871707 28374 consensus_queue.cc:237] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:18:17.874377 28761 raft_consensus.cc:1261] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:18:17.874768 28307 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ede24a27608a4cf196dd46f85c4163e1. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:17.875545 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:17.875571 535 consensus_queue.cc:1035] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:18:17.878010 978 raft_consensus.cc:2932] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:18:17.878670 28761 raft_consensus.cc:2932] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } }
I20240503 08:18:17.880888 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ede24a27608a4cf196dd46f85c4163e1 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:18:17.882663 28226 catalog_manager.cc:5484] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 6 to 7, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:18:17.933722 28309 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:18:18.017828 846 consensus_queue.cc:579] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Leader has been unable to successfully communicate with peer bea2145b8e24436392b18ec3cfe90b93 for more than 300 seconds (300.312s)
I20240503 08:18:18.021324 1009 ts_tablet_manager.cc:902] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:18:18.021976 28374 consensus_queue.cc:237] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:18:18.023042 1009 tablet_copy_client.cc:308] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:18:18.023890 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet 09156806b0744752969b6a574e24b50a from peer 716ffbb2a00b4f43b03e5cedcc9d302a ({username='slave'} at 127.20.107.130:41829)
I20240503 08:18:18.024046 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet 09156806b0744752969b6a574e24b50a from peer 716ffbb2a00b4f43b03e5cedcc9d302a at {username='slave'} at 127.20.107.130:41829: session id = 716ffbb2a00b4f43b03e5cedcc9d302a-09156806b0744752969b6a574e24b50a
I20240503 08:18:18.026273 30414 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 08:18:18.026811 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:18.027160 535 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:18:18.028239 28655 tablet_copy_source_session.cc:205] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 5 blocks and 1 log segments
W20240503 08:18:18.029894 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): Couldn't send request to peer 716ffbb2a00b4f43b03e5cedcc9d302a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:18.030182 978 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:18:18.031140 1009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09156806b0744752969b6a574e24b50a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:18.034358 30414 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:18:18.038467 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 48e31346107a453a81921b23b244e7ff with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:18:18.038663 1009 tablet_copy_client.cc:783] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Starting download of 5 data blocks...
I20240503 08:18:18.041796 28228 catalog_manager.cc:5484] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 reported cstate change: config changed from index 9 to 10, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:18:18.046849 1026 ts_tablet_manager.cc:902] T d60a162155504975a39c20eff88341e1 P c8375219addc48fa99ac474ed1c9aef9: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:18:18.048331 1009 tablet_copy_client.cc:647] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:18:18.049211 1026 tablet_copy_client.cc:308] T d60a162155504975a39c20eff88341e1 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:18:18.050243 1009 tablet_copy_client.cc:523] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:18:18.055799 1009 tablet_bootstrap.cc:492] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:18:18.061009 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet d60a162155504975a39c20eff88341e1 from peer c8375219addc48fa99ac474ed1c9aef9 ({username='slave'} at 127.20.107.133:58789)
I20240503 08:18:18.061182 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet d60a162155504975a39c20eff88341e1 from peer c8375219addc48fa99ac474ed1c9aef9 at {username='slave'} at 127.20.107.133:58789: session id = c8375219addc48fa99ac474ed1c9aef9-d60a162155504975a39c20eff88341e1
I20240503 08:18:18.064100 28655 tablet_copy_source_session.cc:205] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:18:18.065748 1026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60a162155504975a39c20eff88341e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:18.071846 1026 tablet_copy_client.cc:783] T d60a162155504975a39c20eff88341e1 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 5 data blocks...
I20240503 08:18:18.079910 1026 tablet_copy_client.cc:647] T d60a162155504975a39c20eff88341e1 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:18:18.081669 1026 tablet_copy_client.cc:523] T d60a162155504975a39c20eff88341e1 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:18:18.084298 1009 tablet_bootstrap.cc:492] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:18:18.084555 1026 tablet_bootstrap.cc:492] T d60a162155504975a39c20eff88341e1 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap starting.
I20240503 08:18:18.084661 1009 tablet_bootstrap.cc:492] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap complete.
I20240503 08:18:18.084961 1009 ts_tablet_manager.cc:1367] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.008s
I20240503 08:18:18.085577 1009 raft_consensus.cc:348] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:18:18.085808 1009 raft_consensus.cc:729] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: LEARNER
I20240503 08:18:18.086016 1009 consensus_queue.cc:260] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:18:18.086573 1009 ts_tablet_manager.cc:1398] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:18:18.087263 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session 716ffbb2a00b4f43b03e5cedcc9d302a-09156806b0744752969b6a574e24b50a received from {username='slave'} at 127.20.107.130:41829
I20240503 08:18:18.087368 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session 716ffbb2a00b4f43b03e5cedcc9d302a-09156806b0744752969b6a574e24b50a on tablet 09156806b0744752969b6a574e24b50a with peer 716ffbb2a00b4f43b03e5cedcc9d302a
I20240503 08:18:18.101964 1026 tablet_bootstrap.cc:492] T d60a162155504975a39c20eff88341e1 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:18:18.102388 1026 tablet_bootstrap.cc:492] T d60a162155504975a39c20eff88341e1 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap complete.
I20240503 08:18:18.102659 1026 ts_tablet_manager.cc:1367] T d60a162155504975a39c20eff88341e1 P c8375219addc48fa99ac474ed1c9aef9: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.004s
I20240503 08:18:18.103257 1026 raft_consensus.cc:348] T d60a162155504975a39c20eff88341e1 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:18:18.103462 1026 raft_consensus.cc:729] T d60a162155504975a39c20eff88341e1 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8375219addc48fa99ac474ed1c9aef9, State: Initialized, Role: LEARNER
I20240503 08:18:18.103658 1026 consensus_queue.cc:260] T d60a162155504975a39c20eff88341e1 P c8375219addc48fa99ac474ed1c9aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:18:18.104450 1026 ts_tablet_manager.cc:1398] T d60a162155504975a39c20eff88341e1 P c8375219addc48fa99ac474ed1c9aef9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:18:18.105238 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session c8375219addc48fa99ac474ed1c9aef9-d60a162155504975a39c20eff88341e1 received from {username='slave'} at 127.20.107.133:58789
I20240503 08:18:18.105369 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session c8375219addc48fa99ac474ed1c9aef9-d60a162155504975a39c20eff88341e1 on tablet d60a162155504975a39c20eff88341e1 with peer c8375219addc48fa99ac474ed1c9aef9
I20240503 08:18:18.136648 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling UndoDeltaBlockGCOp(d60a162155504975a39c20eff88341e1): 4921 bytes on disk
I20240503 08:18:18.137307 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: UndoDeltaBlockGCOp(d60a162155504975a39c20eff88341e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240503 08:18:18.151629 1009 ts_tablet_manager.cc:902] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:18:18.152925 1009 tablet_copy_client.cc:308] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:18:18.153625 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet b4ceeeb3495e4a8fac6e13e431c64af4 from peer 716ffbb2a00b4f43b03e5cedcc9d302a ({username='slave'} at 127.20.107.130:41829)
I20240503 08:18:18.153815 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet b4ceeeb3495e4a8fac6e13e431c64af4 from peer 716ffbb2a00b4f43b03e5cedcc9d302a at {username='slave'} at 127.20.107.130:41829: session id = 716ffbb2a00b4f43b03e5cedcc9d302a-b4ceeeb3495e4a8fac6e13e431c64af4
I20240503 08:18:18.156836 28655 tablet_copy_source_session.cc:205] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:18:18.158591 1009 ts_tablet_manager.cc:1875] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:18:18.163710 1009 ts_tablet_manager.cc:1888] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:18:18.163988 1009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ceeeb3495e4a8fac6e13e431c64af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:18.167030 1009 tablet_copy_client.cc:783] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Starting download of 5 data blocks...
I20240503 08:18:18.169546 1026 ts_tablet_manager.cc:902] T 0caf86850b3841ea811b0a6ab19ae330 P c8375219addc48fa99ac474ed1c9aef9: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
W20240503 08:18:18.170773 28440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:18:18.175107 1009 tablet_copy_client.cc:647] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:18:18.177052 1009 tablet_copy_client.cc:523] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:18:18.179917 1009 tablet_bootstrap.cc:492] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:18:18.184373 1026 tablet_copy_client.cc:308] T 0caf86850b3841ea811b0a6ab19ae330 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:18:18.190999 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 0caf86850b3841ea811b0a6ab19ae330 from peer c8375219addc48fa99ac474ed1c9aef9 ({username='slave'} at 127.20.107.133:53699)
I20240503 08:18:18.191183 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 0caf86850b3841ea811b0a6ab19ae330 from peer c8375219addc48fa99ac474ed1c9aef9 at {username='slave'} at 127.20.107.133:53699: session id = c8375219addc48fa99ac474ed1c9aef9-0caf86850b3841ea811b0a6ab19ae330
I20240503 08:18:18.194878 28785 tablet_copy_source_session.cc:205] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:18:18.196396 1026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caf86850b3841ea811b0a6ab19ae330. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:18.199065 1009 tablet_bootstrap.cc:492] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:18:18.199198 1050 ts_tablet_manager.cc:902] T 173376257c4946b5a801a27e4dffe90d P b29361eb4afa4ab3af7d8019d8a5042d: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:18:18.199448 1009 tablet_bootstrap.cc:492] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap complete.
F20240503 08:18:18.199534 1050 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-09156806b0744752969b6a574e24b50a.queue_time_us, tablet-download-pool-09156806b0744752969b6a574e24b50a.queue_time_us) (tablet-download-pool-09156806b0744752969b6a574e24b50a.run_cpu_time_us, tablet-download-pool-09156806b0744752969b6a574e24b50a.run_cpu_time_us) (tablet-download-pool-09156806b0744752969b6a574e24b50a.run_wall_time_us, tablet-download-pool-09156806b0744752969b6a574e24b50a.run_wall_time_us) (tablet-download-pool-173376257c4946b5a801a27e4dffe90d.queue_time_us, tablet-download-pool-173376257c4946b5a801a27e4dffe90d.queue_time_us) (tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.queue_time_us, tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.queue_time_us) (tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.run_cpu_time_us, tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.run_cpu_time_us) (tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.run_wall_time_us, tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.run_wall_time_us) (tablet-download-pool-320d1e880958427a88e93321171cb36e.queue_time_us, tablet-download-pool-320d1e880958427a88e93321171cb36e.queue_time_us) (tablet-download-pool-320d1e880958427a88e93321171cb36e.run_cpu_time_us, tablet-download-pool-320d1e880958427a88e93321171cb36e.run_cpu_time_us) (tablet-download-pool-320d1e880958427a88e93321171cb36e.run_wall_time_us, tablet-download-pool-320d1e880958427a88e93321171cb36e.run_wall_time_us) (tablet-download-pool-3bfa8537bc8a4950b53fbf3c36d5ecf3.queue_time_us, tablet-download-pool-3bfa8537bc8a4950b53fbf3c36d5ecf3.queue_time_us) (tablet-download-pool-3bfa8537bc8a4950b53fbf3c36d5ecf3.run_cpu_time_us, tablet-download-pool-3bfa8537bc8a4950b53fbf3c36d5ecf3.run_cpu_time_us) (tablet-download-pool-3bfa8537bc8a4950b53fbf3c36d5ecf3.run_wall_time_us, tablet-download-pool-3bfa8537bc8a4950b53fbf3c36d5ecf3.run_wall_time_us) (tablet-download-pool-9b3b8b80ebd44f27899f97211266c7b2.queue_time_us, tablet-download-pool-9b3b8b80ebd44f27899f97211266c7b2.queue_time_us) (tablet-download-pool-9b3b8b80ebd44f27899f97211266c7b2.run_cpu_time_us, tablet-download-pool-9b3b8b80ebd44f27899f97211266c7b2.run_cpu_time_us) (tablet-download-pool-9b3b8b80ebd44f27899f97211266c7b2.run_wall_time_us, tablet-download-pool-9b3b8b80ebd44f27899f97211266c7b2.run_wall_time_us) (tablet-download-pool-a20489bf65774902af38d228ee966ffe.queue_time_us, tablet-download-pool-a20489bf65774902af38d228ee966ffe.queue_time_us) (tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_cpu_time_us, tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_cpu_time_us) (tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_wall_time_us, tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_wall_time_us) (tablet-download-pool-ac392ff7965a405f96efc777aca4f0bb.queue_time_us, tablet-download-pool-ac392ff7965a405f96efc777aca4f0bb.queue_time_us) (tablet-download-pool-ac392ff7965a405f96efc777aca4f0bb.run_cpu_time_us, tablet-download-pool-ac392ff7965a405f96efc777aca4f0bb.run_cpu_time_us) (tablet-download-pool-ac392ff7965a405f96efc777aca4f0bb.run_wall_time_us, tablet-download-pool-ac392ff7965a405f96efc777aca4f0bb.run_wall_time_us) (tablet-download-pool-b0d07a6278d545e69bf7d2012109d1f4.queue_time_us, tablet-download-pool-b0d07a6278d545e69bf7d2012109d1f4.queue_time_us) (tablet-download-pool-b0d07a6278d545e69bf7d2012109d1f4.run_cpu_time_us, tablet-download-pool-b0d07a6278d545e69bf7d2012109d1f4.run_cpu_time_us) (tablet-download-pool-b0d07a6278d545e69bf7d2012109d1f4.run_wall_time_us, tablet-download-pool-b0d07a6278d545e69bf7d2012109d1f4.run_wall_time_us) (tablet-download-pool-b4ceeeb3495e4a8fac6e13e431c64af4.queue_time_us, tablet-download-pool-b4ceeeb3495e4a8fac6e13e431c64af4.queue_time_us) (tablet-download-pool-b4ceeeb3495e4a8fac6e13e431c64af4.run_cpu_time_us, tablet-download-pool-b4ceeeb3495e4a8fac6e13e431c64af4.run_cpu_time_us) (tablet-download-pool-b4ceeeb3495e4a8fac6e13e431c64af4.run_wall_time_us, tablet-download-pool-b4ceeeb3495e4a8fac6e13e431c64af4.run_wall_time_us) (tablet-download-pool-b5736ee8065344ea8124fdf759035216.queue_time_us, tablet-download-pool-b5736ee8065344ea8124fdf759035216.queue_time_us) (tablet-download-pool-b5736ee8065344ea8124fdf759035216.run_cpu_time_us, tablet-download-pool-b5736ee8065344ea8124fdf759035216.run_cpu_time_us) (tablet-download-pool-b5736ee8065344ea8124fdf759035216.run_wall_time_us, tablet-download-pool-b5736ee8065344ea8124fdf759035216.run_wall_time_us) (tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.queue_time_us, tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.queue_time_us) (tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.run_cpu_time_us, tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.run_cpu_time_us) (tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.run_wall_time_us, tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.run_wall_time_us) (tablet-download-pool-d60a162155504975a39c20eff88341e1.queue_time_us, tablet-download-pool-d60a162155504975a39c20eff88341e1.queue_time_us) (tablet-download-pool-d60a162155504975a39c20eff88341e1.run_cpu_time_us, tablet-download-pool-d60a162155504975a39c20eff88341e1.run_cpu_time_us) (tablet-download-pool-d60a162155504975a39c20eff88341e1.run_wall_time_us, tablet-download-pool-d60a162155504975a39c20eff88341e1.run_wall_time_us) (tablet-download-pool-da529c8f8de241d38678d1596b9402ac.queue_time_us, tablet-download-pool-da529c8f8de241d38678d1596b9402ac.queue_time_us) (tablet-download-pool-da529c8f8de241d38678d1596b9402ac.run_cpu_time_us, tablet-download-pool-da529c8f8de241d38678d1596b9402ac.run_cpu_time_us) (tablet-download-pool-da529c8f8de241d38678d1596b9402ac.run_wall_time_us, tablet-download-pool-da529c8f8de241d38678d1596b9402ac.run_wall_time_us) (tablet-download-pool-e9997e8da99c4f45a63bc3db521b33ba.queue_time_us, tablet-download-pool-e9997e8da99c4f45a63bc3db521b33ba.queue_time_us) (tablet-download-pool-e9997e8da99c4f45a63bc3db521b33ba.run_cpu_time_us, tablet-download-pool-e9997e8da99c4f45a63bc3db521b33ba.run_cpu_time_us) (tablet-download-pool-e9997e8da99c4f45a63bc3db521b33ba.run_wall_time_us, tablet-download-pool-e9997e8da99c4f45a63bc3db521b33ba.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240503 08:18:18.199721 1009 ts_tablet_manager.cc:1367] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.020s user 0.020s sys 0.000s
*** Check failure stack trace: ***
I20240503 08:18:18.200485 1009 raft_consensus.cc:348] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:18:18.200697 1009 raft_consensus.cc:729] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: LEARNER
I20240503 08:18:18.200923 1009 consensus_queue.cc:260] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:18:18.201633 1009 ts_tablet_manager.cc:1398] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
@ 0x7fe8505bddcd google::LogMessage::Fail() at ??:0
I20240503 08:18:18.202394 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session 716ffbb2a00b4f43b03e5cedcc9d302a-b4ceeeb3495e4a8fac6e13e431c64af4 received from {username='slave'} at 127.20.107.130:41829
I20240503 08:18:18.202523 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session 716ffbb2a00b4f43b03e5cedcc9d302a-b4ceeeb3495e4a8fac6e13e431c64af4 on tablet b4ceeeb3495e4a8fac6e13e431c64af4 with peer 716ffbb2a00b4f43b03e5cedcc9d302a
I20240503 08:18:18.203073 1026 tablet_copy_client.cc:783] T 0caf86850b3841ea811b0a6ab19ae330 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 5 data blocks...
@ 0x7fe8505c1b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fe8505bd7cc google::LogMessage::Flush() at ??:0
@ 0x7fe8505bef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe8532fe742 kudu::TraceMetrics::InternName() at ??:0
I20240503 08:18:18.212651 847 consensus_queue.cc:579] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Leader has been unable to successfully communicate with peer bea2145b8e24436392b18ec3cfe90b93 for more than 300 seconds (300.457s)
@ 0x7fe8532caf8b kudu::ThreadPool::ThreadPool() at ??:0
I20240503 08:18:18.217085 1026 tablet_copy_client.cc:647] T 0caf86850b3841ea811b0a6ab19ae330 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:18:18.219383 1026 tablet_copy_client.cc:523] T 0caf86850b3841ea811b0a6ab19ae330 P c8375219addc48fa99ac474ed1c9aef9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7fe8532c4cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 08:18:18.224090 1026 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap starting.
@ 0x7fe8627188de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 08:18:18.235263 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(b4ceeeb3495e4a8fac6e13e431c64af4): 4923 bytes on disk
@ 0x7fe862730daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 08:18:18.236491 28502 consensus_queue.cc:237] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } }
I20240503 08:18:18.241508 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(b4ceeeb3495e4a8fac6e13e431c64af4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
I20240503 08:18:18.242523 30672 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:18:18.242650 30279 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
@ 0x7fe86285b480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 08:18:18.243206 847 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:18:18.243829 847 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:18:18.244983 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(09156806b0744752969b6a574e24b50a): 4838 bytes on disk
I20240503 08:18:18.246414 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(09156806b0744752969b6a574e24b50a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
@ 0x7fe862893a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fe862876981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fe86287670e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fe8532ceb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe8532d219c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240503 08:18:18.271373 1053 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
@ 0x7fe8532d2019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240503 08:18:18.277729 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a20489bf65774902af38d228ee966ffe with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
@ 0x7fe8532ad01e kudu::Thread::SuperviseThread() at ??:0
I20240503 08:18:18.281754 28226 catalog_manager.cc:5484] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:18:18.282145 30676 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P b29361eb4afa4ab3af7d8019d8a5042d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
I20240503 08:18:18.285187 30279 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } }
@ 0x7fe8566326db start_thread at ??:0
I20240503 08:18:18.288251 28502 consensus_queue.cc:237] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
@ 0x7fe84f47071f clone at ??:0
W20240503 08:18:18.291990 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a20489bf65774902af38d228ee966ffe. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:18.299919 1066 ts_tablet_manager.cc:902] T ede24a27608a4cf196dd46f85c4163e1 P c8375219addc48fa99ac474ed1c9aef9: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
F20240503 08:18:18.300340 1066 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.queue_time_us, tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.queue_time_us) (tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.run_cpu_time_us, tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.run_cpu_time_us) (tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.run_wall_time_us, tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.run_wall_time_us) (tablet-download-pool-564fb1b51abd4f8da0372ec99e3c599c.queue_time_us, tablet-download-pool-564fb1b51abd4f8da0372ec99e3c599c.queue_time_us) (tablet-download-pool-564fb1b51abd4f8da0372ec99e3c599c.run_cpu_time_us, tablet-download-pool-564fb1b51abd4f8da0372ec99e3c599c.run_cpu_time_us) (tablet-download-pool-564fb1b51abd4f8da0372ec99e3c599c.run_wall_time_us, tablet-download-pool-564fb1b51abd4f8da0372ec99e3c599c.run_wall_time_us) (tablet-download-pool-585ba864cc384e7f9db90e9d4c47e82c.queue_time_us, tablet-download-pool-585ba864cc384e7f9db90e9d4c47e82c.queue_time_us) (tablet-download-pool-585ba864cc384e7f9db90e9d4c47e82c.run_cpu_time_us, tablet-download-pool-585ba864cc384e7f9db90e9d4c47e82c.run_cpu_time_us) (tablet-download-pool-585ba864cc384e7f9db90e9d4c47e82c.run_wall_time_us, tablet-download-pool-585ba864cc384e7f9db90e9d4c47e82c.run_wall_time_us) (tablet-download-pool-6e9537f30aaf45adbc2cc4a784009de7.queue_time_us, tablet-download-pool-6e9537f30aaf45adbc2cc4a784009de7.queue_time_us) (tablet-download-pool-6e9537f30aaf45adbc2cc4a784009de7.run_cpu_time_us, tablet-download-pool-6e9537f30aaf45adbc2cc4a784009de7.run_cpu_time_us) (tablet-download-pool-6e9537f30aaf45adbc2cc4a784009de7.run_wall_time_us, tablet-download-pool-6e9537f30aaf45adbc2cc4a784009de7.run_wall_time_us) (tablet-download-pool-745d736d43244d4f9063f61de77cb473.queue_time_us, tablet-download-pool-745d736d43244d4f9063f61de77cb473.queue_time_us) (tablet-download-pool-745d736d43244d4f9063f61de77cb473.run_cpu_time_us, tablet-download-pool-745d736d43244d4f9063f61de77cb473.run_cpu_time_us) (tablet-download-pool-745d736d43244d4f9063f61de77cb473.run_wall_time_us, tablet-download-pool-745d736d43244d4f9063f61de77cb473.run_wall_time_us) (tablet-download-pool-a20489bf65774902af38d228ee966ffe.queue_time_us, tablet-download-pool-a20489bf65774902af38d228ee966ffe.queue_time_us) (tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_cpu_time_us, tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_cpu_time_us) (tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_wall_time_us, tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_wall_time_us) (tablet-download-pool-a8a9adc3712d436c95505a3216e7607c.queue_time_us, tablet-download-pool-a8a9adc3712d436c95505a3216e7607c.queue_time_us) (tablet-download-pool-a8a9adc3712d436c95505a3216e7607c.run_cpu_time_us, tablet-download-pool-a8a9adc3712d436c95505a3216e7607c.run_cpu_time_us) (tablet-download-pool-a8a9adc3712d436c95505a3216e7607c.run_wall_time_us, tablet-download-pool-a8a9adc3712d436c95505a3216e7607c.run_wall_time_us) (tablet-download-pool-a8b4e398d34443b4a440f9ba48555d84.queue_time_us, tablet-download-pool-a8b4e398d34443b4a440f9ba48555d84.queue_time_us) (tablet-download-pool-a8b4e398d34443b4a440f9ba48555d84.run_cpu_time_us, tablet-download-pool-a8b4e398d34443b4a440f9ba48555d84.run_cpu_time_us) (tablet-download-pool-a8b4e398d34443b4a440f9ba48555d84.run_wall_time_us, tablet-download-pool-a8b4e398d34443b4a440f9ba48555d84.run_wall_time_us) (tablet-download-pool-ac392ff7965a405f96efc777aca4f0bb.queue_time_us, tablet-download-pool-ac392ff7965a405f96efc777aca4f0bb.queue_time_us) (tablet-download-pool-ac392ff7965a405f96efc777aca4f0bb.run_cpu_time_us, tablet-download-pool-ac392ff7965a405f96efc777aca4f0bb.run_cpu_time_us) (tablet-download-pool-ac392ff7965a405f96efc777aca4f0bb.run_wall_time_us, tablet-download-pool-ac392ff7965a405f96efc777aca4f0bb.run_wall_time_us) (tablet-download-pool-b0d07a6278d545e69bf7d2012109d1f4.queue_time_us, tablet-download-pool-b0d07a6278d545e69bf7d2012109d1f4.queue_time_us) (tablet-download-pool-b0d07a6278d545e69bf7d2012109d1f4.run_cpu_time_us, tablet-download-pool-b0d07a6278d545e69bf7d2012109d1f4.run_cpu_time_us) (tablet-download-pool-b0d07a6278d545e69bf7d2012109d1f4.run_wall_time_us, tablet-download-pool-b0d07a6278d545e69bf7d2012109d1f4.run_wall_time_us) (tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.queue_time_us, tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.queue_time_us) (tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.run_cpu_time_us, tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.run_cpu_time_us) (tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.run_wall_time_us, tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.run_wall_time_us) (tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.queue_time_us, tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.queue_time_us) (tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.run_cpu_time_us, tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.run_cpu_time_us) (tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.run_wall_time_us, tablet-download-pool-c6e1b9f104ea471cab7527bd56433327.run_wall_time_us) (tablet-download-pool-d44335d8be7245b1bb52b2eab9e42c18.queue_time_us, tablet-download-pool-d44335d8be7245b1bb52b2eab9e42c18.queue_time_us) (tablet-download-pool-d44335d8be7245b1bb52b2eab9e42c18.run_cpu_time_us, tablet-download-pool-d44335d8be7245b1bb52b2eab9e42c18.run_cpu_time_us) (tablet-download-pool-d44335d8be7245b1bb52b2eab9e42c18.run_wall_time_us, tablet-download-pool-d44335d8be7245b1bb52b2eab9e42c18.run_wall_time_us) (tablet-download-pool-d60a162155504975a39c20eff88341e1.queue_time_us, tablet-download-pool-d60a162155504975a39c20eff88341e1.queue_time_us) (tablet-download-pool-d60a162155504975a39c20eff88341e1.run_cpu_time_us, tablet-download-pool-d60a162155504975a39c20eff88341e1.run_cpu_time_us) (tablet-download-pool-d60a162155504975a39c20eff88341e1.run_wall_time_us, tablet-download-pool-d60a162155504975a39c20eff88341e1.run_wall_time_us) (tablet-download-pool-ede24a27608a4cf196dd46f85c4163e1.queue_time_us, tablet-download-pool-ede24a27608a4cf196dd46f85c4163e1.queue_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240503 08:18:18.300599 1026 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:18:18.303135 1026 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P c8375219addc48fa99ac474ed1c9aef9: Bootstrap complete.
I20240503 08:18:18.303468 1026 ts_tablet_manager.cc:1367] T 0caf86850b3841ea811b0a6ab19ae330 P c8375219addc48fa99ac474ed1c9aef9: Time spent bootstrapping tablet: real 0.079s user 0.031s sys 0.005s
I20240503 08:18:18.304472 1026 raft_consensus.cc:348] T 0caf86850b3841ea811b0a6ab19ae330 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:18:18.304781 1026 raft_consensus.cc:729] T 0caf86850b3841ea811b0a6ab19ae330 P c8375219addc48fa99ac474ed1c9aef9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8375219addc48fa99ac474ed1c9aef9, State: Initialized, Role: LEARNER
I20240503 08:18:18.305063 1026 consensus_queue.cc:260] T 0caf86850b3841ea811b0a6ab19ae330 P c8375219addc48fa99ac474ed1c9aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } }
I20240503 08:18:18.305768 1026 ts_tablet_manager.cc:1398] T 0caf86850b3841ea811b0a6ab19ae330 P c8375219addc48fa99ac474ed1c9aef9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 08:18:18.306516 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session c8375219addc48fa99ac474ed1c9aef9-0caf86850b3841ea811b0a6ab19ae330 received from {username='slave'} at 127.20.107.133:53699
I20240503 08:18:18.306638 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session c8375219addc48fa99ac474ed1c9aef9-0caf86850b3841ea811b0a6ab19ae330 on tablet 0caf86850b3841ea811b0a6ab19ae330 with peer c8375219addc48fa99ac474ed1c9aef9
*** Check failure stack trace: ***
@ 0x7fe2f77c3dcd google::LogMessage::Fail() at ??:0
I20240503 08:18:18.314078 30279 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
@ 0x7fe2f77c7b93 google::LogMessage::SendToLog() at ??:0
I20240503 08:18:18.314698 1053 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
@ 0x7fe2f77c37cc google::LogMessage::Flush() at ??:0
I20240503 08:18:18.318219 30280 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
@ 0x7fe2f77c4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe2fa504742 kudu::TraceMetrics::InternName() at ??:0
I20240503 08:18:18.337705 28226 catalog_manager.cc:5484] T a20489bf65774902af38d228ee966ffe P c8375219addc48fa99ac474ed1c9aef9 reported cstate change: config changed from index 12 to 13, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
@ 0x7fe2fa4d0f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fe2fa4cacd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 08:18:18.361312 1062 raft_consensus.cc:2932] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
@ 0x7fe30991e8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fe309936daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fe309a61480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 08:18:18.375604 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a20489bf65774902af38d228ee966ffe with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
@ 0x7fe309a99a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fe309a7c981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240503 08:18:18.389735 30327 maintenance_manager.cc:419] P c8375219addc48fa99ac474ed1c9aef9: Scheduling UndoDeltaBlockGCOp(0caf86850b3841ea811b0a6ab19ae330): 4583 bytes on disk
@ 0x7fe309a7c70e std::_Function_handler<>::_M_invoke() at ??:0
I20240503 08:18:18.390606 30218 maintenance_manager.cc:643] P c8375219addc48fa99ac474ed1c9aef9: UndoDeltaBlockGCOp(0caf86850b3841ea811b0a6ab19ae330) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fe2fa4d4b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe2fa4d819c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe2fa4d8019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fe2fa4b301e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe2fd8386db start_thread at ??:0
@ 0x7fe2f667671f clone at ??:0
I20240503 08:18:18.416549 28502 raft_consensus.cc:1203] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240503 08:18:18.481182 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a20489bf65774902af38d228ee966ffe: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:18.489437 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 08:18:18.497846 1009 ts_tablet_manager.cc:902] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:18:18.529841 1009 tablet_copy_client.cc:308] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:18:18.539281 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet 48e31346107a453a81921b23b244e7ff from peer 716ffbb2a00b4f43b03e5cedcc9d302a ({username='slave'} at 127.20.107.130:58957)
I20240503 08:18:18.539467 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet 48e31346107a453a81921b23b244e7ff from peer 716ffbb2a00b4f43b03e5cedcc9d302a at {username='slave'} at 127.20.107.130:58957: session id = 716ffbb2a00b4f43b03e5cedcc9d302a-48e31346107a453a81921b23b244e7ff
I20240503 08:18:18.544198 28394 tablet_copy_source_session.cc:205] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:18:18.545645 1009 ts_tablet_manager.cc:1875] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240503 08:18:18.549538 28440 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:18:18.553069 1009 ts_tablet_manager.cc:1888] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240503 08:18:18.553365 1009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e31346107a453a81921b23b244e7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:18.557713 1009 tablet_copy_client.cc:783] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Starting download of 5 data blocks...
I20240503 08:18:18.564742 1009 tablet_copy_client.cc:647] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:18:18.567060 1009 tablet_copy_client.cc:523] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:18:18.571669 1009 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:18:18.599467 1009 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:18:18.599946 1009 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap complete.
I20240503 08:18:18.600234 1009 ts_tablet_manager.cc:1367] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.004s
I20240503 08:18:18.601198 1009 raft_consensus.cc:348] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:18:18.601509 1009 raft_consensus.cc:729] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: LEARNER
I20240503 08:18:18.601750 1009 consensus_queue.cc:260] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:18:18.602445 1009 ts_tablet_manager.cc:1398] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 08:18:18.603384 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session 716ffbb2a00b4f43b03e5cedcc9d302a-48e31346107a453a81921b23b244e7ff received from {username='slave'} at 127.20.107.130:58957
I20240503 08:18:18.603519 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session 716ffbb2a00b4f43b03e5cedcc9d302a-48e31346107a453a81921b23b244e7ff on tablet 48e31346107a453a81921b23b244e7ff with peer 716ffbb2a00b4f43b03e5cedcc9d302a
I20240503 08:18:18.630774 28502 raft_consensus.cc:1203] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
W20240503 08:18:18.727970 30347 connection.cc:535] client connection to 127.20.107.136:42389 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:18:18.727994 28699 connection.cc:535] client connection to 127.20.107.136:42389 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:18:18.728132 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 9 similar messages]
W20240503 08:18:18.728240 30347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:18:18.728691 28569 connection.cc:535] client connection to 127.20.107.136:42389 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:18:18.734187 28439 connection.cc:535] client connection to 127.20.107.136:42389 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:18:18.736549 1076 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:18.735092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.107.136:42389 (local address 127.20.107.129:48163)
0503 08:18:18.735247 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:18.735269 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:18.735379 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:18.735972 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":42}
W20240503 08:18:18.736770 1075 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:18.730005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.107.136:42389 (local address 127.20.107.131:37991)
0503 08:18:18.730422 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:18.730446 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:18.730592 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:18.736313 (+ 5721us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":273}
W20240503 08:18:18.736987 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:18:18.737107 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.737251 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.737356 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.737462 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.737828 29224 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:18.735572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.107.136:42389 (local address 127.20.107.130:39915)
0503 08:18:18.735937 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:18.735953 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:18.736052 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:18.737400 (+ 1348us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":265}
W20240503 08:18:18.741552 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.741739 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.741856 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.741981 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.742100 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.745949 1142 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:18.735630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.107.136:42389 (local address 127.20.107.132:41687)
0503 08:18:18.744004 (+ 8374us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:18.745267 (+ 1263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":8263,"thread_start_us":9577,"threads_started":1}
W20240503 08:18:18.746150 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.746325 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.747081 30347 consensus_peers.cc:485] T 9b3b8b80ebd44f27899f97211266c7b2 P 21d21a5b596b4d48a17a1fb54a972844 -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:18.749953 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(48e31346107a453a81921b23b244e7ff): 4838 bytes on disk
I20240503 08:18:18.750476 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(48e31346107a453a81921b23b244e7ff) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240503 08:18:18.752419 28438 connection.cc:535] client connection to 127.20.107.133:35121 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:18:18.752702 28307 connection.cc:535] client connection to 127.20.107.133:35121 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:18:18.752964 28568 connection.cc:535] client connection to 127.20.107.133:35121 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:18:18.753590 28698 connection.cc:535] client connection to 127.20.107.133:35121 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:18:18.753870 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.753984 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.754175 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.754199 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.754299 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:18.754380 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.754458 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.754740 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.754854 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.754916 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.766606 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.781996 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:18.791308 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
I20240503 08:18:18.877612 901 raft_consensus.cc:1050] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a to VOTER
I20240503 08:18:18.878309 901 consensus_queue.cc:237] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } }
I20240503 08:18:18.880906 28502 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:18:18.880973 28761 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:18:18.881692 1082 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:18:18.882061 1083 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 08:18:18.882342 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:18.884950 1083 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:18:18.885651 28502 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:18:18.890064 28761 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:18:18.894083 28226 catalog_manager.cc:5484] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 9 to 10, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
W20240503 08:18:18.903430 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:18.909682 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 46 similar messages]
I20240503 08:18:18.911330 1151 ts_tablet_manager.cc:902] T a20489bf65774902af38d228ee966ffe P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:18:18.912757 1151 tablet_copy_client.cc:308] T a20489bf65774902af38d228ee966ffe P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:18:18.917729 28524 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet a20489bf65774902af38d228ee966ffe from peer 21d21a5b596b4d48a17a1fb54a972844 ({username='slave'} at 127.20.107.134:46597)
I20240503 08:18:18.917870 28524 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet a20489bf65774902af38d228ee966ffe from peer 21d21a5b596b4d48a17a1fb54a972844 at {username='slave'} at 127.20.107.134:46597: session id = 21d21a5b596b4d48a17a1fb54a972844-a20489bf65774902af38d228ee966ffe
I20240503 08:18:18.920152 28524 tablet_copy_source_session.cc:205] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:18:18.921718 1151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20489bf65774902af38d228ee966ffe. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:18:18.926115 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:18.926616 1151 tablet_copy_client.cc:783] T a20489bf65774902af38d228ee966ffe P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 5 data blocks...
I20240503 08:18:18.932637 1151 tablet_copy_client.cc:647] T a20489bf65774902af38d228ee966ffe P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Starting download of 1 WAL segments...
I20240503 08:18:18.934515 1151 tablet_copy_client.cc:523] T a20489bf65774902af38d228ee966ffe P 21d21a5b596b4d48a17a1fb54a972844: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:18:18.937332 1151 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap starting.
I20240503 08:18:18.959640 1151 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:18:18.960126 1151 tablet_bootstrap.cc:492] T a20489bf65774902af38d228ee966ffe P 21d21a5b596b4d48a17a1fb54a972844: Bootstrap complete.
I20240503 08:18:18.960469 1151 ts_tablet_manager.cc:1367] T a20489bf65774902af38d228ee966ffe P 21d21a5b596b4d48a17a1fb54a972844: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.000s
I20240503 08:18:18.961308 1151 raft_consensus.cc:348] T a20489bf65774902af38d228ee966ffe P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:18:18.961611 1151 raft_consensus.cc:729] T a20489bf65774902af38d228ee966ffe P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d21a5b596b4d48a17a1fb54a972844, State: Initialized, Role: LEARNER
I20240503 08:18:18.961824 1151 consensus_queue.cc:260] T a20489bf65774902af38d228ee966ffe P 21d21a5b596b4d48a17a1fb54a972844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
I20240503 08:18:18.962586 1151 ts_tablet_manager.cc:1398] T a20489bf65774902af38d228ee966ffe P 21d21a5b596b4d48a17a1fb54a972844: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 08:18:18.963351 28524 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session 21d21a5b596b4d48a17a1fb54a972844-a20489bf65774902af38d228ee966ffe received from {username='slave'} at 127.20.107.134:46597
I20240503 08:18:18.963491 28524 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session 21d21a5b596b4d48a17a1fb54a972844-a20489bf65774902af38d228ee966ffe on tablet a20489bf65774902af38d228ee966ffe with peer 21d21a5b596b4d48a17a1fb54a972844
I20240503 08:18:18.967813 30460 maintenance_manager.cc:419] P 21d21a5b596b4d48a17a1fb54a972844: Scheduling UndoDeltaBlockGCOp(a20489bf65774902af38d228ee966ffe): 4838 bytes on disk
I20240503 08:18:18.968438 30351 maintenance_manager.cc:643] P 21d21a5b596b4d48a17a1fb54a972844: UndoDeltaBlockGCOp(a20489bf65774902af38d228ee966ffe) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240503 08:18:18.999930 978 consensus_queue.cc:579] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Leader has been unable to successfully communicate with peer bea2145b8e24436392b18ec3cfe90b93 for more than 300 seconds (300.005s)
I20240503 08:18:19.003518 28374 consensus_queue.cc:237] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:18:19.005565 30414 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:18:19.005548 28630 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:18:19.006157 1173 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:18:19.006476 535 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:18:19.009056 535 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:18:19.009811 30414 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:18:19.010346 28630 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:18:19.012967 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d13c33b8712e4c488abb99b73ff42b86 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:18:19.015298 28232 catalog_manager.cc:5484] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 8 to 9, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:18:19.020651 28374 consensus_queue.cc:237] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
I20240503 08:18:19.022632 28630 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:18:19.023108 30414 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:18:19.023162 535 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 08:18:19.023695 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:19.023787 1173 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:18:19.025833 535 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:18:19.027153 30414 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:18:19.029103 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d13c33b8712e4c488abb99b73ff42b86 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:18:19.031088 28630 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:18:19.031464 28228 catalog_manager.cc:5484] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 9 to 10, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:18:19.119729 28502 raft_consensus.cc:1203] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
W20240503 08:18:19.226505 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
I20240503 08:18:19.249775 1083 raft_consensus.cc:1050] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a to VOTER
I20240503 08:18:19.250470 1083 consensus_queue.cc:237] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } }
I20240503 08:18:19.252357 28502 raft_consensus.cc:1261] T b4ceeeb3495e4a8fac6e13e431c64af4 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:18:19.252990 1083 consensus_queue.cc:1035] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:18:19.253440 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:19.253602 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:19.417461 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
I20240503 08:18:19.424599 30414 raft_consensus.cc:1203] T a20489bf65774902af38d228ee966ffe P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20240503 08:18:19.534346 1052 consensus_queue.cc:579] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Leader has been unable to successfully communicate with peer bea2145b8e24436392b18ec3cfe90b93 for more than 300 seconds (300.384s)
I20240503 08:18:19.538363 28502 consensus_queue.cc:237] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:18:19.540131 28761 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:18:19.540279 30414 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:18:19.540663 1062 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:18:19.540856 1052 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:18:19.543382 1065 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:18:19.543973 30414 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:18:19.544056 28761 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:18:19.546515 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b5736ee8065344ea8124fdf759035216 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:18:19.548233 28232 catalog_manager.cc:5484] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 10 to 11, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:18:19.554538 28502 consensus_queue.cc:237] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
W20240503 08:18:19.554854 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b5736ee8065344ea8124fdf759035216: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
I20240503 08:18:19.556769 28761 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 08:18:19.556993 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:19.557089 30414 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:18:19.557332 1065 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:18:19.557564 1052 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:18:19.559501 1062 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:18:19.560210 28761 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:18:19.560207 30414 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:18:19.562608 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b5736ee8065344ea8124fdf759035216 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:18:19.564808 28232 catalog_manager.cc:5484] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 11 to 12, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:18:19.633965 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
I20240503 08:18:19.705576 535 raft_consensus.cc:1050] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a to VOTER
I20240503 08:18:19.706377 535 consensus_queue.cc:237] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } }
I20240503 08:18:19.708801 28502 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:18:19.708892 30414 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:18:19.709461 1237 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:18:19.709764 28309 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:19.709820 535 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:18:19.712419 535 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:18:19.713083 30414 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:18:19.717679 28502 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:18:19.718268 28228 catalog_manager.cc:5484] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 10 to 11, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:18:19.724463 28374 consensus_queue.cc:237] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } }
I20240503 08:18:19.726267 30414 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:18:19.726397 28502 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:18:19.726871 1173 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:18:19.727757 1173 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:18:19.729012 535 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:18:19.729849 30414 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 21d21a5b596b4d48a17a1fb54a972844 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:18:19.730571 28502 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:18:19.732856 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48e31346107a453a81921b23b244e7ff with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:18:19.733894 28232 catalog_manager.cc:5484] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 11 to 12, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:18:19.880654 901 consensus_queue.cc:579] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Leader has been unable to successfully communicate with peer bea2145b8e24436392b18ec3cfe90b93 for more than 300 seconds (300.243s)
I20240503 08:18:19.885205 28630 consensus_queue.cc:237] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:18:19.887177 28374 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 3 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 08:18:19.887256 28502 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 3 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 08:18:19.887948 1083 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:18:19.888180 901 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:18:19.893566 901 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 5 to 7, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } }
I20240503 08:18:19.894138 28502 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 5 to 7, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } }
I20240503 08:18:19.898053 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba833ce23c4c4e81bc1a81125a1208c9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:18:19.899812 28228 catalog_manager.cc:5484] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 5 to 7, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New cstate: current_term: 3 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } }
I20240503 08:18:19.905920 28374 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 5 to 7, VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } }
I20240503 08:18:19.906222 28630 consensus_queue.cc:237] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } }
W20240503 08:18:19.909420 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
I20240503 08:18:19.910529 1052 raft_consensus.cc:1050] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 to VOTER
W20240503 08:18:19.910929 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:19.911410 1052 consensus_queue.cc:237] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:18:19.911592 28502 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 3 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240503 08:18:19.912549 1083 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:18:19.915818 28374 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 3 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240503 08:18:19.916536 901 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 7 to 8, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:18:19.917227 28502 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:18:19.922760 28217 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ba833ce23c4c4e81bc1a81125a1208c9 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:18:19.923525 30414 raft_consensus.cc:1261] T a20489bf65774902af38d228ee966ffe P 21d21a5b596b4d48a17a1fb54a972844 [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:18:19.921594 28228 catalog_manager.cc:5484] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 7 to 8, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New cstate: current_term: 3 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
I20240503 08:18:19.924453 1062 consensus_queue.cc:1035] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 08:18:19.926069 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:19.926100 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:19.927762 901 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:18:19.929669 28374 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: NON_VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: true } } }
W20240503 08:18:20.037464 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
I20240503 08:18:20.055626 1173 consensus_queue.cc:579] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Leader has been unable to successfully communicate with peer bea2145b8e24436392b18ec3cfe90b93 for more than 300 seconds (300.438s)
I20240503 08:18:20.059890 28374 consensus_queue.cc:237] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } }
I20240503 08:18:20.062330 28630 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20240503 08:18:20.062784 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:20.063086 1173 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:18:20.065515 535 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:18:20.066218 28630 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:18:20.068710 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d44335d8be7245b1bb52b2eab9e42c18 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:18:20.070616 28228 catalog_manager.cc:5484] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 8 to 10, NON_VOTER bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135) evicted. New cstate: current_term: 2 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } }
I20240503 08:18:20.076444 28374 consensus_queue.cc:237] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } }
W20240503 08:18:20.078620 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d44335d8be7245b1bb52b2eab9e42c18. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:20.078967 28630 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:18:20.079471 978 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:18:20.080744 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:18:20.081843 535 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:18:20.082417 28630 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:18:20.085990 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d44335d8be7245b1bb52b2eab9e42c18 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:18:20.088435 28232 catalog_manager.cc:5484] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 10 to 11, NON_VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) added. New cstate: current_term: 2 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } }
I20240503 08:18:20.241406 28222 maintenance_manager.cc:419] P e8f5acc8744f4dccbfd654a6d7dab5bf: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 10322 bytes on disk
I20240503 08:18:20.245697 28221 maintenance_manager.cc:643] P e8f5acc8744f4dccbfd654a6d7dab5bf: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.004s user 0.001s sys 0.003s Metrics: {"dirs.queue_time_us":404,"dirs.run_cpu_time_us":100,"dirs.run_wall_time_us":99,"thread_start_us":256,"threads_started":1}
W20240503 08:18:20.359712 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:20.456931 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:20.522953 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:20.536633 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
I20240503 08:18:20.607972 1317 ts_tablet_manager.cc:902] T d44335d8be7245b1bb52b2eab9e42c18 P 21d21a5b596b4d48a17a1fb54a972844: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
F20240503 08:18:20.608321 1317 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.queue_time_us, tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.queue_time_us) (tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.run_cpu_time_us, tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.run_cpu_time_us) (tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.run_wall_time_us, tablet-download-pool-0caf86850b3841ea811b0a6ab19ae330.run_wall_time_us) (tablet-download-pool-173376257c4946b5a801a27e4dffe90d.queue_time_us, tablet-download-pool-173376257c4946b5a801a27e4dffe90d.queue_time_us) (tablet-download-pool-173376257c4946b5a801a27e4dffe90d.run_cpu_time_us, tablet-download-pool-173376257c4946b5a801a27e4dffe90d.run_cpu_time_us) (tablet-download-pool-173376257c4946b5a801a27e4dffe90d.run_wall_time_us, tablet-download-pool-173376257c4946b5a801a27e4dffe90d.run_wall_time_us) (tablet-download-pool-28e42c3244f949c7b7343bf587894ac5.queue_time_us, tablet-download-pool-28e42c3244f949c7b7343bf587894ac5.queue_time_us) (tablet-download-pool-28e42c3244f949c7b7343bf587894ac5.run_cpu_time_us, tablet-download-pool-28e42c3244f949c7b7343bf587894ac5.run_cpu_time_us) (tablet-download-pool-28e42c3244f949c7b7343bf587894ac5.run_wall_time_us, tablet-download-pool-28e42c3244f949c7b7343bf587894ac5.run_wall_time_us) (tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.queue_time_us, tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.queue_time_us) (tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.run_cpu_time_us, tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.run_cpu_time_us) (tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.run_wall_time_us, tablet-download-pool-2d113e8de02f447b9da0a1cb03686cd2.run_wall_time_us) (tablet-download-pool-3bfa8537bc8a4950b53fbf3c36d5ecf3.queue_time_us, tablet-download-pool-3bfa8537bc8a4950b53fbf3c36d5ecf3.queue_time_us) (tablet-download-pool-3bfa8537bc8a4950b53fbf3c36d5ecf3.run_cpu_time_us, tablet-download-pool-3bfa8537bc8a4950b53fbf3c36d5ecf3.run_cpu_time_us) (tablet-download-pool-3bfa8537bc8a4950b53fbf3c36d5ecf3.run_wall_time_us, tablet-download-pool-3bfa8537bc8a4950b53fbf3c36d5ecf3.run_wall_time_us) (tablet-download-pool-48e31346107a453a81921b23b244e7ff.queue_time_us, tablet-download-pool-48e31346107a453a81921b23b244e7ff.queue_time_us) (tablet-download-pool-48e31346107a453a81921b23b244e7ff.run_cpu_time_us, tablet-download-pool-48e31346107a453a81921b23b244e7ff.run_cpu_time_us) (tablet-download-pool-48e31346107a453a81921b23b244e7ff.run_wall_time_us, tablet-download-pool-48e31346107a453a81921b23b244e7ff.run_wall_time_us) (tablet-download-pool-4babf0ea9b954a23b1b1c4e403dce0bd.queue_time_us, tablet-download-pool-4babf0ea9b954a23b1b1c4e403dce0bd.queue_time_us) (tablet-download-pool-4babf0ea9b954a23b1b1c4e403dce0bd.run_cpu_time_us, tablet-download-pool-4babf0ea9b954a23b1b1c4e403dce0bd.run_cpu_time_us) (tablet-download-pool-4babf0ea9b954a23b1b1c4e403dce0bd.run_wall_time_us, tablet-download-pool-4babf0ea9b954a23b1b1c4e403dce0bd.run_wall_time_us) (tablet-download-pool-585ba864cc384e7f9db90e9d4c47e82c.queue_time_us, tablet-download-pool-585ba864cc384e7f9db90e9d4c47e82c.queue_time_us) (tablet-download-pool-585ba864cc384e7f9db90e9d4c47e82c.run_cpu_time_us, tablet-download-pool-585ba864cc384e7f9db90e9d4c47e82c.run_cpu_time_us) (tablet-download-pool-585ba864cc384e7f9db90e9d4c47e82c.run_wall_time_us, tablet-download-pool-585ba864cc384e7f9db90e9d4c47e82c.run_wall_time_us) (tablet-download-pool-9b3b8b80ebd44f27899f97211266c7b2.queue_time_us, tablet-download-pool-9b3b8b80ebd44f27899f97211266c7b2.queue_time_us) (tablet-download-pool-9b3b8b80ebd44f27899f97211266c7b2.run_cpu_time_us, tablet-download-pool-9b3b8b80ebd44f27899f97211266c7b2.run_cpu_time_us) (tablet-download-pool-9b3b8b80ebd44f27899f97211266c7b2.run_wall_time_us, tablet-download-pool-9b3b8b80ebd44f27899f97211266c7b2.run_wall_time_us) (tablet-download-pool-a20489bf65774902af38d228ee966ffe.queue_time_us, tablet-download-pool-a20489bf65774902af38d228ee966ffe.queue_time_us) (tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_cpu_time_us, tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_cpu_time_us) (tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_wall_time_us, tablet-download-pool-a20489bf65774902af38d228ee966ffe.run_wall_time_us) (tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.queue_time_us, tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.queue_time_us) (tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.run_cpu_time_us, tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.run_cpu_time_us) (tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.run_wall_time_us, tablet-download-pool-b2ebb0b8475b4ed9a09711a87867de08.run_wall_time_us) (tablet-download-pool-b5736ee8065344ea8124fdf759035216.queue_time_us, tablet-download-pool-b5736ee8065344ea8124fdf759035216.queue_time_us) (tablet-download-pool-b5736ee8065344ea8124fdf759035216.run_cpu_time_us, tablet-download-pool-b5736ee8065344ea8124fdf759035216.run_cpu_time_us) (tablet-download-pool-b5736ee8065344ea8124fdf759035216.run_wall_time_us, tablet-download-pool-b5736ee8065344ea8124fdf759035216.run_wall_time_us) (tablet-download-pool-d13c33b8712e4c488abb99b73ff42b86.queue_time_us, tablet-download-pool-d13c33b8712e4c488abb99b73ff42b86.queue_time_us) (tablet-download-pool-d13c33b8712e4c488abb99b73ff42b86.run_cpu_time_us, tablet-download-pool-d13c33b8712e4c488abb99b73ff42b86.run_cpu_time_us) (tablet-download-pool-d13c33b8712e4c488abb99b73ff42b86.run_wall_time_us, tablet-download-pool-d13c33b8712e4c488abb99b73ff42b86.run_wall_time_us) (tablet-download-pool-d44335d8be7245b1bb52b2eab9e42c18.queue_time_us, tablet-download-pool-d44335d8be7245b1bb52b2eab9e42c18.queue_time_us) (tablet-download-pool-e9997e8da99c4f45a63bc3db521b33ba.queue_time_us, tablet-download-pool-e9997e8da99c4f45a63bc3db521b33ba.queue_time_us) (tablet-download-pool-e9997e8da99c4f45a63bc3db521b33ba.run_cpu_time_us, tablet-download-pool-e9997e8da99c4f45a63bc3db521b33ba.run_cpu_time_us) (tablet-download-pool-e9997e8da99c4f45a63bc3db521b33ba.run_wall_time_us, tablet-download-pool-e9997e8da99c4f45a63bc3db521b33ba.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f4557175dcd google::LogMessage::Fail() at ??:0
@ 0x7f4557179b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f45571757cc google::LogMessage::Flush() at ??:0
@ 0x7f4557176f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4559eb6742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f4559e82f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f4559e7ccd6 kudu::ThreadPoolBuilder::Build() at ??:0
W20240503 08:18:20.627969 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
@ 0x7f45692d08de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f45692e8daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f4569413480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f456944ba8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f456942e981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f456942e70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f4559e86b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4559e8a19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f4559e8a019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f4559e6501e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f455d1ea6db start_thread at ??:0
@ 0x7f455602871f clone at ??:0
W20240503 08:18:20.696214 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:20.708400 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d44335d8be7245b1bb52b2eab9e42c18: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:20.789759 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:20.851321 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:20.894217 28699 connection.cc:535] client connection to 127.20.107.134:34587 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:18:20.894364 28308 connection.cc:535] client connection to 127.20.107.134:34587 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:18:20.895531 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:20.895677 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:20.895982 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:20.896096 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:20.902209 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:20.906857 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:20.928220 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:20.945483 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:20.947858 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:20.972622 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:20.995769 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:21.011046 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:21.032493 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:21.054965 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:21.056802 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:21.057600 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:21.059170 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:21.091848 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:21.120036 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:21.135095 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:21.143417 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:21.177848 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:18:21.200289 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:21.205688 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:21.219467 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:21.289548 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:21.315402 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:21.317890 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:21.333206 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:21.348701 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:21.382076 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:21.403087 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:21.564015 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:21.575492 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:21.577939 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:21.623248 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:21.654067 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:21.697122 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:21.739944 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d60a162155504975a39c20eff88341e1: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:21.813205 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:21.820484 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:18:21.920964 1368 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:18:21.921144 1368 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:18:21.921736 1368 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:18:21.923245 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:21.923429 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:18:21.923507 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:18:21.923741 1368 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:18:22.156414 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:22.183122 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:22.217780 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:22.219641 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:22.395749 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:22.511879 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:22.589746 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:22.606243 28219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240503 08:18:22.646008 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:22.729079 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:22.824412 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:22.843055 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:22.905385 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:22.918527 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:22.959666 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:23.066958 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:23.090994 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.130831 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:23.171087 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 08:18:23.191718 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:23.222397 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.269171 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:23.270181 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.272464 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.282539 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:23.285923 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d13c33b8712e4c488abb99b73ff42b86: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:23.298046 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:23.310346 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.350564 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.379076 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:23.413733 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.450666 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.486191 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:23.487810 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:23.488296 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:23.542838 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 08:18:23.543752 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:23.557372 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:23.564649 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.577953 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:23.584156 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.625188 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:23.632843 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:18:23.691675 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.699513 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:18:23.747633 1368 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:18:23.747843 1368 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:18:23.748493 1368 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:18:23.749423 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240503 08:18:23.750308 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:23.750500 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:18:23.750586 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:18:23.750828 1368 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:18:23.773370 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.782987 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.815290 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.825404 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:23.889827 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.920492 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240503 08:18:23.953245 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.960237 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:23.964587 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:24.014103 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:24.032373 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:24.038640 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:24.115885 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:24.128830 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:24.196416 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:24.203763 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:24.223815 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:24.391589 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d44335d8be7245b1bb52b2eab9e42c18: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:24.531292 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:24.552547 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:24.647581 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:24.815341 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:24.826174 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:25.008311 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:25.030928 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:25.046386 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:25.238376 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:25.247781 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:25.348701 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:25.373977 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:25.404855 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:25.483244 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:25.515723 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:25.539407 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:25.625239 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:25.634332 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:25.637316 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:25.711388 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:25.715324 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:25.760576 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:25.763875 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:18:25.822180 1368 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:18:25.822381 1368 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:18:25.823027 1368 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:18:25.824568 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:25.824780 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:18:25.824909 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:18:25.825135 1368 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:18:25.825958 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:25.838579 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d60a162155504975a39c20eff88341e1: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:25.868917 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:25.883576 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:25.954092 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:25.975594 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:25.978358 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:25.987571 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:25.996162 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:26.012445 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:26.066134 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.165802 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:26.186466 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.194717 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.233357 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.241230 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:26.242106 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:26.249451 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:26.273187 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.284124 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.354702 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.362751 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:18:26.382761 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.435396 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.464581 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.514431 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.543363 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.549767 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:26.569679 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:26.613035 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.640492 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.709404 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.785831 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.795615 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:26.800695 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:26.939867 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:27.035869 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:27.115896 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:27.140548 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:27.263288 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:27.307564 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:27.388290 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d13c33b8712e4c488abb99b73ff42b86: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:27.448055 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:27.449404 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:27.546881 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:27.592463 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:27.790545 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:27.817875 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:27.840085 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:27.913347 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:27.926263 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:27.998754 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.009124 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:28.013185 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.033134 28219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 08:18:28.045351 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:28.045977 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:28.194078 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 125 similar messages]
W20240503 08:18:28.197893 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:28.236676 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.237562 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:28.266175 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:28.297585 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:28.308985 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:28.346194 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.398909 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.413952 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:28.450364 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:28.515980 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d44335d8be7245b1bb52b2eab9e42c18: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:28.575080 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.621529 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:18:28.622519 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.626940 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:28.639403 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.643133 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.654577 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:28.661881 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:28.679347 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.688162 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.700120 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.734231 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.746196 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:28.747988 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:28.765652 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:18:28.766386 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.783448 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.784549 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:18:28.788733 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:28.872097 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:28.945935 28568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 08:18:28.965451 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:28.980739 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:29.025110 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:29.137429 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:29.150118 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:29.169744 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:29.189487 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:29.259361 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:29.271229 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:29.310619 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:29.317620 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:29.361234 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
I20240503 08:18:29.541798 1930 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:18:29.542084 1930 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:18:29.542690 1930 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:18:29.544296 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:29.544494 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:18:29.544585 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:18:29.544900 1930 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:18:29.648502 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:29.754225 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:29.861063 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:29.880066 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:29.994246 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:29.995484 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:30.146435 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:30.226181 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:30.292695 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:30.328896 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:30.329092 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:30.356539 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:30.362993 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:30.366981 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:30.510713 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:30.577230 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:30.577998 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:30.581594 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:30.700698 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:30.703892 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:30.713707 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:30.728780 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:30.754316 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:30.786911 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:30.837906 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:30.898669 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:30.923064 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:30.968952 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:30.988379 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:30.999190 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.003329 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:31.042579 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.077967 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.094502 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.096134 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
I20240503 08:18:31.101913 1994 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:18:31.102099 1994 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:18:31.102699 1994 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:18:31.104358 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:31.104553 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:18:31.104648 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:18:31.104948 1994 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:18:31.121793 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:31.142244 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:31.184113 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:31.231397 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.253110 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.299273 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:31.324748 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:31.338269 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:18:31.378723 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.382961 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.439076 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.519114 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.530212 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:31.535585 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.594300 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.660276 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.716439 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.725922 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:31.729168 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.819993 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.827628 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.883297 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:31.936373 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:32.136198 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:32.230100 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:32.262091 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:32.311653 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:32.313370 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:32.485774 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:32.642442 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:32.679747 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:32.822512 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:32.830408 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:32.845533 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:32.891255 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:32.894222 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:32.957419 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.046520 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.125751 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:33.146756 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:33.199554 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.209193 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.220276 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.233294 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:33.263950 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:18:33.288360 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.292531 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:33.358011 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:33.379840 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:33.409200 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:33.416358 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:33.492596 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.504833 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.542088 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:33.543344 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:33.578470 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.588495 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:33.595826 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:33.638518 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:33.662281 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.713780 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:18:33.714669 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:33.716732 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:33.745406 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.758744 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:18:33.785439 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.791952 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 08:18:33.853866 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.864135 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:33.865198 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.866719 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.975556 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 08:18:33.982300 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:33.995368 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:34.032799 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:34.036000 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:34.050658 28219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240503 08:18:34.051970 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d60a162155504975a39c20eff88341e1: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:34.098723 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:34.099018 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:34.152088 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:34.187188 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:34.200278 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:18:34.256306 1994 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:18:34.256556 1994 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:18:34.257362 1994 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:18:34.258985 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:34.259196 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:18:34.259286 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:18:34.259537 1994 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:18:34.289426 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:34.297745 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:34.327544 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:34.389605 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:34.504216 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:34.546382 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:34.629243 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:34.691934 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:34.754434 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:34.937958 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:34.979207 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:34.988906 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:35.062059 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:35.227586 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:35.256266 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:35.349071 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:35.435387 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:35.461037 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:35.494051 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:35.510213 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:35.517125 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:35.628300 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d13c33b8712e4c488abb99b73ff42b86: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:35.634819 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:35.683094 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:35.714327 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:35.767414 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:35.779414 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:35.802271 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:35.809461 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:35.847470 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:35.871783 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:35.960065 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:35.988796 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:35.999994 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.047353 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:36.058974 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:36.080819 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:36.108065 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:36.112421 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.146574 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:36.172832 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:36.196892 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:36.196934 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.204083 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:18:36.266067 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.316220 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:36.349263 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.363077 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.368048 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.401201 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.441664 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.580433 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.587532 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.601791 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.605105 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:36.678620 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.686623 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:36.730399 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.744501 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d44335d8be7245b1bb52b2eab9e42c18: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:36.763748 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.779222 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:36.928790 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:37.021065 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:37.021247 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:37.024595 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:37.025722 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:37.066669 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:37.106595 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:37.444206 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:37.510561 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:37.587553 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:37.624540 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:37.675122 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:37.729414 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:37.764240 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:37.901108 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:37.996294 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:38.002118 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:38.021232 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:38.150653 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:38.160929 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:38.188331 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:38.216508 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:38.251104 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:38.318181 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:18:38.340500 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:38.343318 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:38.356729 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:38.362447 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:38.363535 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:38.378470 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:38.406396 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:38.411378 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:38.419020 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:38.439119 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:38.442206 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:38.521526 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:38.563349 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:38.563987 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:38.569272 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:38.606568 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:38.646950 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:38.670339 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:38.677219 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:38.726471 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:38.728097 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:18:38.766572 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240503 08:18:38.809062 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:38.821365 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:18:38.863746 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:38.868060 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:38.896337 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:38.971503 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:39.004653 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 08:18:39.015739 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:39.020670 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:39.024199 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:39.028470 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:39.042873 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:39.061057 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:39.247912 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:39.279649 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:39.291195 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:39.456156 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:39.548504 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:39.686228 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:39.686587 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:39.701239 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:39.713804 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:39.794271 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:39.808669 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:39.842298 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:40.038905 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:40.224164 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:40.242434 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:40.267099 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:40.326272 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:40.405092 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:40.432579 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:40.459369 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:40.542943 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:40.571983 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:40.622781 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:40.655170 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:40.733731 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:40.758787 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:40.761229 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:40.763986 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:40.841701 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:40.872395 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:40.884428 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:40.910743 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:40.929349 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:40.936945 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:40.967998 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:40.986979 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.022080 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.050825 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.064411 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:41.073869 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.104087 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.114905 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:41.131880 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:41.136816 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:41.195392 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.293491 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:41.336966 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.451727 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:41.452358 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.460956 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:41.477792 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.483940 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.494648 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:18:41.505378 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.621170 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.621569 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:41.621681 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.772377 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.792157 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.883522 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:41.936094 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:42.048044 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:42.054996 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:42.072100 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:42.160692 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:42.313347 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:42.340423 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:42.395809 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:42.496768 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:42.626765 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:42.672901 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:42.715468 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:42.736508 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:42.849476 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:42.855924 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:42.891880 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:42.982776 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:43.058588 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:43.062340 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:43.097936 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:43.104740 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:43.110682 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:43.190351 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:43.213241 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:43.238237 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:43.248493 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:43.262286 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:43.315097 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:43.322643 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 08:18:43.323387 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:43.330173 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:43.357774 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:43.409898 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:43.439890 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:43.445865 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:43.457116 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:43.457263 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:43.463755 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:43.486646 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:43.571947 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:43.575574 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:43.646188 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:43.686496 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:43.706701 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:43.776974 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:18:43.798523 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:43.808995 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:43.849334 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240503 08:18:43.865398 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:43.871650 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:43.895114 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:43.900184 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:18:44.088641 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 08:18:44.130165 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:44.135336 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:44.227306 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:44.259699 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:44.262907 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:44.373554 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:44.402984 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:44.436731 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:44.441427 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:44.480229 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:44.558585 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:44.588887 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:44.653288 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:44.661204 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:44.673251 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:44.722837 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:44.742362 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:45.056638 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:45.094216 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:45.184826 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:45.195856 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:45.225363 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:45.226683 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:45.263936 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:45.274184 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:45.312070 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:45.369264 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:45.411582 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:45.484143 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:45.515806 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:45.556046 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:45.641561 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:45.699185 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:45.718868 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:45.724457 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:45.724970 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:45.807196 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:45.813068 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:45.827055 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:45.839059 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:45.854612 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:45.944090 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:45.955099 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:46.003492 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:46.006675 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:46.015869 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:46.057729 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:46.144924 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:46.151247 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:46.171020 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:46.172855 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:46.274621 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:46.334276 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:46.337301 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:46.346102 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:46.354086 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:18:46.434916 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:46.438575 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:46.537601 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:46.624070 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:46.659042 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:46.801844 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:46.847656 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:46.865337 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:46.894819 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:46.924489 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:46.939252 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:46.978477 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:46.982570 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:47.034111 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:47.074460 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:47.098570 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:47.190932 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:47.297847 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:47.325863 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:47.328622 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:47.455349 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:47.619742 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:47.623935 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:47.655509 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:47.655686 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:47.731635 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:47.769315 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:47.793251 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:47.795542 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:47.875685 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:47.907893 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:48.013288 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:48.069969 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:48.101624 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:48.118829 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:48.137359 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:48.196226 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:48.204145 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:48.275650 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:48.298120 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:48.363375 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:48.369625 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:48.396222 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:18:48.396999 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:48.419256 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:48.544265 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:48.592150 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:48.596037 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:48.671754 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:48.703809 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:48.714334 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:48.721055 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:48.789191 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:18:48.790721 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:48.794159 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:48.812317 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:48.817786 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:48.851908 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:48.854820 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 08:18:48.855639 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:48.868960 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:48.891561 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:18:48.903282 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:49.015949 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:49.054612 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:49.077775 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:49.262751 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 08:18:49.265669 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:49.289886 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:49.308682 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
I20240503 08:18:49.311657 3050 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:18:49.311853 3050 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:18:49.312633 3050 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:18:49.313992 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:49.314141 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:18:49.314236 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:18:49.314599 3050 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:18:49.318066 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:49.329918 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:49.353462 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:49.424546 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:49.437458 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:49.486292 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:49.548167 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:49.562211 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:49.713292 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:49.756738 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:49.773123 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:49.895592 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:49.924254 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:49.985342 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:50.109606 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:50.119892 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:50.264153 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:50.273752 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:50.329092 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:50.408841 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:50.422340 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:50.424657 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:50.431083 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:50.453461 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:50.466110 28219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:18:50.467314 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d60a162155504975a39c20eff88341e1: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:50.538326 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:50.557801 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:50.604328 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:50.692416 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:50.766182 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:50.807927 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:50.815726 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:50.861673 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:50.869123 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:50.929019 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:50.931684 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:50.953858 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:50.968318 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:51.128337 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:51.128654 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.134202 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.157699 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.187866 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:51.207607 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.214195 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:51.216653 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.238938 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.258285 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.398792 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:18:51.418377 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:51.425367 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:51.551875 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.569977 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:51.582964 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.655685 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.683568 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.690433 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.743734 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.746399 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.750514 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.799482 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:51.830866 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:51.900301 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:51.950891 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:52.031960 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d13c33b8712e4c488abb99b73ff42b86: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:52.101063 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:52.179100 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:52.285363 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:52.358886 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:52.369269 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:52.373385 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:52.381425 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:52.496640 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:52.614140 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:52.685622 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:52.703593 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:52.822700 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:52.868752 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:52.868983 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:52.898952 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:52.898967 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:52.929927 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:52.979173 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:52.997655 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:53.056856 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:53.108426 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:53.131971 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d44335d8be7245b1bb52b2eab9e42c18: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:18:53.138389 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:53.188263 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:53.229100 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:53.244993 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:53.265341 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:53.291621 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:53.351244 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:53.401266 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:53.417368 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:18:53.418329 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:53.428488 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:53.461166 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:53.487957 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:53.564323 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:53.622848 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:53.633306 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:53.676601 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:53.686071 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:53.705472 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:53.714248 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:53.759245 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:53.771935 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:53.797286 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 08:18:53.814358 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:53.910065 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:18:53.921819 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:53.972213 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:54.033334 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:18:54.117321 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:54.128433 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:54.162670 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:54.163136 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:54.200052 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:54.205369 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:54.279568 28568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 08:18:54.284487 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:54.378703 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:54.454732 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:54.458736 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:54.459582 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:54.497754 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:54.509967 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:54.572536 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:54.607828 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:54.708580 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:54.810074 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:54.892946 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:55.001667 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:55.021009 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:55.114245 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:55.133590 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:55.179978 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:55.204965 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:55.254015 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:55.254660 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:55.290920 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:55.476475 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:55.506134 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:55.525535 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:55.539664 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:55.602783 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:55.652531 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:55.678645 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:55.684198 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:55.706614 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:55.773029 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:55.800853 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:55.802541 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:55.832381 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:55.949643 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:55.973091 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:55.974659 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:55.994944 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:56.009742 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:56.014020 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:56.025943 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:56.039852 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:56.122663 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:56.200395 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:56.246081 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:56.268774 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:56.277408 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:56.296801 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:56.340880 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:56.369410 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:56.416172 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:56.464176 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:18:56.482398 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:56.569766 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:56.596287 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:56.623625 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:56.657284 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:56.704514 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:56.804239 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:56.959223 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:56.962690 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:56.991545 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:57.004242 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:57.008033 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:57.038187 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:57.319247 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:57.321990 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:57.335016 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:57.336647 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:57.348441 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:57.412078 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:57.469496 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:57.475582 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:57.658154 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:57.878608 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:57.904034 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:57.909863 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:57.918476 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:57.947297 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:57.980309 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:58.025445 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:58.034987 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:58.085911 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:58.166826 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:58.177482 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:58.188364 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:58.188802 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:58.195794 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:58.236979 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:58.286387 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:58.288772 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:58.333792 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:58.405383 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:58.414899 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:58.423424 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:18:58.487299 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:58.521678 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:58.549621 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:58.591518 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:58.593483 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:58.674654 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:58.781064 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:58.820044 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 08:18:58.820924 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:58.831461 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:58.844949 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:58.902539 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:58.935797 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:58.943720 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240503 08:18:58.944746 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:58.947504 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:58.954030 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:59.038376 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:59.069864 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:59.072716 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:59.098883 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:59.195096 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:18:59.298964 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 08:18:59.315438 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:59.344388 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:59.350167 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:59.359679 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:59.404036 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:59.432180 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:59.564247 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:59.627571 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:18:59.703830 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:18:59.727794 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:59.730511 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:59.795497 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:59.858667 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:18:59.886116 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:18:59.906718 3425 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:18:59.906903 3425 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:18:59.907672 3425 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:18:59.909154 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:18:59.909339 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:18:59.909426 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:18:59.909641 3425 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:19:00.075584 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:00.100477 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:00.108489 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:00.122207 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:00.136435 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:00.251633 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:00.361413 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:00.401307 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:00.437748 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:00.447465 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:00.487223 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:00.555183 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:00.581401 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:00.596035 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:00.616068 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:00.723253 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:00.737835 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:00.743458 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:00.750119 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:00.798777 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:00.824206 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:00.919391 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:00.935663 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:00.985761 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:00.996496 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:01.053493 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:01.102764 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:01.103332 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:01.144806 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:01.169420 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:01.210860 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:01.247926 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:01.272959 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:01.308238 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:01.314694 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:01.401569 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:01.402786 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:01.460011 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:01.466593 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:01.476575 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:01.492391 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:01.513737 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:01.537983 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:01.647449 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:01.660037 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:01.701169 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:01.733795 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:19:01.837528 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:01.849684 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:01.870240 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:01.959378 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:02.051734 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:02.080065 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:02.204957 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:02.229106 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:02.263716 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:02.268002 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:02.389775 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:02.424729 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:02.571373 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:02.649595 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:19:02.720252 4130 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:19:02.720417 4130 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:19:02.721084 4130 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:19:02.722877 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:02.723047 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:19:02.723119 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:19:02.723341 4130 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:19:02.797780 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:02.821357 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:02.857064 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:02.898017 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:02.925949 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:02.933109 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:02.951609 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:02.978636 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:03.035382 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:03.043342 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:03.062671 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:03.079329 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:03.092027 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:03.158967 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:03.218437 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:03.222694 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:03.354939 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:03.410483 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:03.430745 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:19:03.431550 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:03.475279 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:03.480005 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:03.487190 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:03.549364 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:03.571123 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:03.573377 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:03.617849 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:03.776477 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:03.793920 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:03.835017 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:03.835922 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:03.837014 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:03.852813 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:03.900328 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:03.925259 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:19:03.926075 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:03.935439 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:03.935882 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:04.000715 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:04.041675 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:04.070044 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:04.097100 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:04.119125 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240503 08:19:04.217108 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:04.251490 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:04.308091 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:04.313560 28568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 08:19:04.361434 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:19:04.369392 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:04.442037 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:04.460718 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:04.490819 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:04.504258 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:04.558305 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:04.649544 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:04.690080 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:04.748415 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:04.881186 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:04.946741 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:05.065436 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:05.071975 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:05.203307 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:05.319456 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:05.343894 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:05.352671 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:05.376273 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:05.389449 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:05.413848 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:05.419732 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:05.449589 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:05.484910 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:05.533761 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:05.541718 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:05.702611 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:05.755025 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:05.776535 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:05.784114 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:05.809113 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:05.872615 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:05.885728 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:05.886199 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:05.907578 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:05.939863 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:05.969300 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:05.985160 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:06.008530 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:06.049530 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:06.053258 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:06.101541 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:06.113394 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:06.157826 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:06.204804 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:06.217520 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:06.283643 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:06.329171 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:06.371217 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:06.386407 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:06.422091 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:06.513093 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:06.556422 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:06.558668 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:06.669888 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:06.805979 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:06.828337 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:06.842525 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:06.921759 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:19:06.926671 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:06.930160 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:06.938922 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:07.039206 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:07.055897 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:07.095180 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:07.137332 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:07.325784 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:07.339725 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:07.399096 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:07.457456 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:07.513456 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:07.525566 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:07.633327 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:07.642205 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:07.673992 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:07.804934 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:07.825435 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:07.852033 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:07.897657 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:07.937623 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:08.053459 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:08.075881 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:08.082693 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:08.090469 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:08.172188 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:08.267073 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:08.289158 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:08.309932 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:08.325580 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:08.361065 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:08.369376 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:08.406356 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:08.434650 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:08.438793 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:08.439154 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:19:08.469432 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:08.570374 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:08.599100 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:08.626305 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:08.627390 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:08.667328 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:08.685292 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:08.715858 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:08.725562 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:19:08.805862 4508 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:19:08.806088 4508 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:19:08.806842 4508 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:19:08.808496 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:08.808662 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:19:08.808751 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:19:08.809010 4508 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:19:08.815037 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:08.835892 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:08.845124 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:08.866396 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:08.960191 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:19:08.977150 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:09.089741 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:09.125089 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:09.130568 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240503 08:19:09.169481 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:09.176491 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:09.186726 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:09.205541 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:09.320535 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 08:19:09.364089 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:09.368435 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:09.431996 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:09.447249 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:19:09.461620 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:09.512111 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:09.524062 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:09.617228 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:09.723735 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:09.774616 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:09.909821 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:09.910292 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:09.952745 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:09.957165 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:10.066964 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:10.164400 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:10.175860 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:10.226552 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:10.297655 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:10.317787 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:10.347538 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:10.359073 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:10.467275 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:10.516299 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:10.533947 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:10.573736 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:10.634002 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:10.645067 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:10.658303 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:10.708817 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:10.760495 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:10.813632 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:10.817912 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:10.866528 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:10.875195 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:10.880548 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:10.894079 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:10.908186 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:10.972193 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:10.984267 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:11.099853 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:11.142449 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:11.166241 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:11.178009 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:11.211323 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:11.222043 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:11.257640 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:11.328811 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:11.357060 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:11.364971 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:11.385818 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:11.520952 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:11.525801 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:11.581828 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:11.610003 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:11.731858 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:19:11.774472 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:11.781100 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:11.811273 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:11.820269 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:11.921312 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:11.956426 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:12.080031 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:12.101255 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:12.178117 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:12.203316 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:12.230664 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:12.240442 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:12.253193 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:12.270210 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:12.294263 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:12.446934 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:12.488777 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:12.629762 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:12.710845 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:12.743847 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:12.747954 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:12.763485 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:12.875363 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:12.875679 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:12.979861 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:13.017891 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:13.073733 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:13.090965 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:13.120219 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:13.140766 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:13.146095 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:13.254266 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:13.280310 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:13.287668 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:13.305874 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:13.352089 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:13.356371 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:13.370188 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:13.375931 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:13.459249 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:13.471035 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:19:13.490530 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:13.609422 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:13.654361 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:13.707444 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:13.713264 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:13.732661 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:13.737522 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:13.743188 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:13.807552 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:13.842490 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:13.852046 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:13.861869 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:13.935609 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:13.990895 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:14.041733 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:14.047102 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:19:14.114912 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:14.144490 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:14.268106 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 08:19:14.286412 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:19:14.297583 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:14.300151 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:14.316126 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:14.325965 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:14.337152 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:14.356936 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 08:19:14.491145 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:14.569620 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:14.572610 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:14.666653 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:14.680171 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:14.702946 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:14.753105 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:14.822115 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:14.842888 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:14.862713 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:14.880537 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:15.070502 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:15.078183 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:15.100422 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:15.212920 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:15.266116 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:15.323722 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:15.438484 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:15.464825 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
I20240503 08:19:15.487735 4963 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:19:15.487954 4963 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:19:15.488607 4963 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:19:15.490064 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:15.490259 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:19:15.490345 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:19:15.490568 4963 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:19:15.534904 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:15.556231 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:15.562494 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:15.562777 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:15.568619 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:15.586338 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:15.623541 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:15.690645 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:15.714008 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:15.762368 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:15.909425 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:15.927515 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:15.936110 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:16.010171 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:16.041827 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:16.042418 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:16.088640 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:16.128187 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:16.162674 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:16.266114 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:16.266708 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:16.284407 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:16.304741 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:16.341893 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:16.447062 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:16.504267 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:16.531379 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:16.545074 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:16.549036 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:16.608645 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:16.673405 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:16.679649 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:16.695881 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:16.741801 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:16.753973 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:16.759917 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:16.773720 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:16.822840 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:19:16.907541 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:16.950501 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:17.042958 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:17.079144 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:17.099119 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:17.102834 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:17.176293 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:17.198722 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:17.221239 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:17.451771 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:17.460358 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:17.490288 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:17.530259 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:17.594084 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:17.616317 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:17.811388 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:17.822691 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:17.908715 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:17.933282 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:17.943111 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:17.943369 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:17.949029 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:17.968266 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:17.979929 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:18.015528 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:18.022975 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:18.037125 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:18.139343 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:18.187520 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:18.254820 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:18.485687 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:18.495386 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:18.500576 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:19:18.513957 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:18.525280 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:18.572383 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:18.630403 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:18.640954 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:18.676752 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:18.698669 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:18.712570 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:18.714269 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:18.740618 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:18.768029 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:18.877996 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:19.009729 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:19.012820 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:19.063163 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:19.095232 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:19.095387 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:19:19.096118 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:19.129851 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:19.145498 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:19.146970 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:19:19.227819 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:19.234724 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:19.236716 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:19.243340 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:19.261777 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:19.306448 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:19.358119 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:19.370265 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 08:19:19.382041 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:19.406140 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240503 08:19:19.553627 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:19.578728 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:19.595196 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:19.595289 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:19.596633 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:19.679318 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:19.683887 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:19.935454 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:19.944675 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:20.111541 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:20.115124 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:20.159732 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:20.172859 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:20.212600 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:20.277621 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:20.318336 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:20.334704 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:20.336042 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:20.361874 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:20.423183 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:20.471458 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:20.497382 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:20.606019 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:20.674117 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:20.701576 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:20.758752 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:20.761955 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:20.766654 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:20.923341 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:20.927546 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:20.974946 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:21.029923 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:21.068998 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:21.120360 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:21.180370 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:21.186254 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:21.192106 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:21.199407 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:21.213780 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:21.274372 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:21.289064 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:21.319108 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:21.446142 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:21.586593 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:21.640470 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:21.667148 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:21.681761 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:21.683362 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:19:21.733822 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:21.741736 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:21.749382 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:21.790251 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:21.804489 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:21.808122 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:21.817310 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:21.833786 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:21.865242 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:21.872375 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:21.999126 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:22.039363 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:22.107393 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:22.177678 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:22.327400 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:22.342756 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:22.404830 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:22.476109 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:22.571970 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:22.622143 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:22.637538 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:22.637681 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:22.686367 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:22.784528 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:22.803942 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:22.821010 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:22.897006 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:22.937937 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:22.941125 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:23.059929 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:23.094771 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:23.099975 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:23.103477 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:23.118971 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:23.174434 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:23.196086 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:23.215404 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:23.252187 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:23.267511 28219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:19:23.268702 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d60a162155504975a39c20eff88341e1: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:19:23.377363 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:23.413465 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:23.431317 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:23.443591 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:23.547695 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:19:23.553025 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:23.573454 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:23.644356 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:23.674679 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:23.714514 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:23.799409 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:23.809813 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:23.857878 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:23.914090 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:23.969470 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:24.003608 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:24.019032 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:24.024200 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:24.032943 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:24.046550 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:19:24.096691 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:19:24.143285 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:24.205941 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:24.234848 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:24.260095 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:24.261076 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:24.268353 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:24.271492 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:24.407276 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 08:19:24.413791 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:19:24.452685 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:24.454332 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:24.478415 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:24.523808 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:24.542456 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:24.579375 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:24.603402 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:24.845952 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:24.847333 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d13c33b8712e4c488abb99b73ff42b86: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:19:24.848029 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:24.853332 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:24.903442 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:25.096411 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:25.103765 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:25.130300 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:25.139919 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:25.159569 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:25.234138 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:25.306895 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:25.329538 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:25.430128 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:25.433763 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:25.502399 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:25.566596 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:25.656605 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:25.657692 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:25.685130 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:25.709790 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:25.714005 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:25.739861 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:25.753784 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:25.793934 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:25.816342 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:25.892112 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:25.926514 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:25.933629 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d44335d8be7245b1bb52b2eab9e42c18: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:19:25.935935 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:25.986517 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:26.076062 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:26.083086 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:26.159132 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:26.164953 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:26.168942 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:26.172506 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:26.316216 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:26.367091 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:26.413833 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:26.426954 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:26.505584 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:26.526111 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:26.623070 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:26.637709 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:26.650807 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:26.652348 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:26.668498 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:26.744328 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:19:26.763252 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:26.787345 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:26.802714 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:26.805536 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:26.900557 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:27.010349 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:27.035499 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:27.046126 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:27.137130 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:27.155920 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:27.200713 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:27.320245 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:27.426429 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:27.495844 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:27.529177 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:27.581218 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:27.607484 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:27.649436 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:27.679973 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:27.689836 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:27.764780 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:27.865743 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:28.011798 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:28.020370 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:28.087603 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:28.093713 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:28.110397 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:28.170554 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:28.172487 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:28.197780 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:28.210933 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:28.251147 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:28.267354 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:28.275605 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:28.304232 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:28.402084 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:28.413076 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:28.415764 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:28.424731 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:28.457558 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:28.489679 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:28.507944 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:28.528435 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:28.562197 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:28.621248 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:19:28.643364 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:28.673492 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:28.762220 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:28.798738 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:28.922770 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:28.948776 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:29.043151 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:29.043335 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:29.074237 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:29.141155 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240503 08:19:29.202404 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:29.205085 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:29.245575 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:29.287030 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:29.364516 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:29.431648 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:29.440107 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 08:19:29.462694 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:29.473975 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:19:29.474923 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:29.475327 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:29.514372 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:29.538616 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:29.543455 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:19:29.557013 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:29.560607 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:29.578037 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:29.582594 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:29.796289 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:29.891249 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:29.934157 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:19:29.975953 5713 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:19:29.976161 5713 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:19:29.976812 5713 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:19:29.978793 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:29.979022 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:19:29.979144 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:19:29.979386 5713 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:19:29.979924 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:29.986268 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:30.035358 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:30.044652 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:30.146813 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:30.149907 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:30.274963 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:30.310704 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:30.389132 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:30.432835 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:30.440476 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:30.519973 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:30.520280 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:30.539461 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:30.570288 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:30.652920 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:30.693953 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:30.703260 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:30.704941 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:30.706894 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:30.807785 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:30.846067 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:30.935144 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:30.993788 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:30.996976 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:31.014317 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:31.105223 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:31.122583 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:31.139029 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:31.143784 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:31.147305 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:31.188257 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:31.245489 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:31.368788 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:31.387076 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:31.389658 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:31.439934 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:31.505793 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:31.525611 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:31.644922 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:31.664098 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:19:31.698647 6190 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:19:31.698843 6190 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:19:31.699509 6190 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:19:31.701066 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:31.701273 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:19:31.701368 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:19:31.701620 6190 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:19:31.741294 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:31.763130 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:19:31.763206 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:31.802238 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:31.834765 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:31.909324 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:31.917110 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:31.919438 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:31.923437 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:31.947492 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:32.001410 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:32.028007 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:32.040177 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:32.061076 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:32.212788 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:32.418843 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:32.453598 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:32.475874 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:32.490936 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:32.496748 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:32.500368 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:32.676108 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:32.782716 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:32.785715 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:32.788611 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:32.817922 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:32.907557 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:32.941519 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:32.990583 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:33.005087 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:33.017683 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:33.051640 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:33.065099 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:33.119694 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:33.169952 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:33.170343 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:19:33.268004 6190 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:19:33.268326 6190 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:19:33.269086 6190 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:19:33.270798 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:33.271001 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:19:33.271106 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:19:33.271399 6190 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:19:33.272228 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:33.291045 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:33.350157 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:33.350589 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:33.379197 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:33.442458 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:33.484771 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:33.514726 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:33.521874 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:33.577715 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:33.580816 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:33.633471 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 08:19:33.634363 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:33.652150 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:33.714255 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:33.751406 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:33.797683 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:33.911799 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:33.925205 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:33.942466 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:33.973387 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:33.983711 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:34.017203 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:34.114212 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:19:34.149616 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:19:34.304504 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:34.315310 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:34.324487 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:34.336827 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:34.368188 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:34.389515 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:34.391629 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:34.414441 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:34.441072 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:34.459357 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 08:19:34.483105 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240503 08:19:34.546416 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:34.607182 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:34.689751 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:34.700454 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:34.703478 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:34.761516 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:34.908814 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:34.953119 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:34.977542 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:34.977728 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:35.071930 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:35.119369 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:35.189770 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:35.195606 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:35.197094 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:35.302512 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:35.325243 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:35.434746 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:35.500540 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:35.528568 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:35.536943 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:35.556452 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:35.557113 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:35.574331 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:35.588749 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:35.709640 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:35.715622 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:35.805246 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:35.811364 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:35.814088 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:35.838214 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:35.871843 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:35.885888 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:35.908941 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:36.072114 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:36.104862 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:36.152179 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:36.178723 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:36.194310 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:36.200894 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:36.201167 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:36.269912 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:36.287667 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:36.302112 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:36.353610 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:36.366798 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:36.386009 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:36.513162 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:36.531265 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:36.582801 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:19:36.592720 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:36.701587 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:19:36.735518 6510 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:19:36.735738 6510 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:19:36.736449 6510 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:19:36.737722 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:36.737890 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:19:36.737984 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:19:36.738250 6510 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:19:36.776855 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:36.896466 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:36.979559 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:37.059693 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.066856 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.069206 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.069418 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:37.105948 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.206264 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:37.210553 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.273150 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.295640 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.324851 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.379649 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:37.387881 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.470829 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.568058 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.618556 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:37.636679 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:37.639765 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:37.779503 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.790450 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.795128 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.812696 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.884240 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.928277 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:37.971035 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:37.994904 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:38.014850 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:38.037341 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:38.153734 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:38.215984 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:38.247169 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:38.325670 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:38.391073 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:38.412490 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:38.413868 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:38.425825 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:38.426546 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:38.465574 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:38.481940 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:38.524044 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:38.533362 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:38.578058 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:38.601316 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:38.610994 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:38.611634 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:38.612035 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:38.649670 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:38.652465 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:38.668604 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:19:38.769449 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:38.804731 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:38.989571 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:39.000977 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:39.040895 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:39.143555 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:39.166268 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240503 08:19:39.167507 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:39.182343 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:19:39.186120 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:39.254736 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:39.286329 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:39.346194 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:39.481491 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:39.506101 28568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 114 similar messages]
W20240503 08:19:39.514206 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 08:19:39.528107 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:39.553889 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:39.564822 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:39.572719 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:39.608423 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:39.608515 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:39.620417 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:39.641454 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:39.653271 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:39.784003 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:39.896279 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:39.916177 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:40.086438 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:40.146863 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:40.167347 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:40.178156 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:40.189494 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:40.196370 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:40.254066 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:40.277040 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:40.362645 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:40.379500 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:40.412842 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:40.526495 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:40.577886 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:40.603215 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:40.626964 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:40.666589 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:40.697259 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:40.781801 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:40.787740 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:40.793251 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:40.808168 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:40.914315 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:40.939368 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:40.954064 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:41.038086 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:41.052071 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:41.077986 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:41.082458 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:41.096994 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:41.100633 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:41.113826 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:41.158775 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:41.199380 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:41.241071 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:41.357028 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:41.357553 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:41.407121 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:41.420277 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:41.436195 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:41.506625 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:41.537493 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:41.641304 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:41.645298 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:19:41.680127 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:41.740438 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:41.848114 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:41.897789 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:41.945204 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:42.001197 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:42.023682 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:42.096328 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:42.136474 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:42.188915 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:42.277431 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:42.311035 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:42.459347 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:42.471709 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:42.483023 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:42.488538 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:42.495230 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:42.562323 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:42.706938 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:42.730722 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:42.734938 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:42.837715 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:42.855011 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:42.885998 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:42.910212 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:42.958546 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:42.986317 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:43.052742 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:43.068907 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:43.072177 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:43.140054 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:43.153020 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:43.197006 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:43.245218 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:43.327239 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:43.342726 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:43.382447 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:43.456405 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:43.538159 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:43.538166 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:43.567440 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:43.583287 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:43.596035 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:43.601794 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:43.619714 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:43.626559 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:43.651665 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:43.686735 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 08:19:43.687716 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:43.824054 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:43.918020 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:43.956255 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:43.987407 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:43.998693 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:44.045223 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:44.060396 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:44.104295 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:44.104704 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:44.107656 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:44.191397 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:19:44.238560 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:44.246977 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:19:44.360190 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:44.395035 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:44.506085 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 08:19:44.617815 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:19:44.619211 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:44.668741 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:44.679808 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:44.724124 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:44.799149 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:44.885571 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:45.014837 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:45.042205 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:45.096390 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:45.109493 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:45.174225 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:45.187850 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:45.190145 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:45.243479 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:45.250802 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:45.286213 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:45.367523 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:45.431525 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:45.453858 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:45.466217 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:45.508165 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:45.527335 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:45.538424 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:45.603225 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:45.609081 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:45.609699 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:45.621701 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:45.622012 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:45.630518 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:45.639453 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:45.754426 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:45.834092 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:45.870745 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:45.884502 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:45.890722 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:45.907125 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:45.922063 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:45.961855 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:45.986502 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:46.119668 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:46.138604 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:46.182607 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:46.319918 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:46.363731 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:46.408900 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:46.469028 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:46.479013 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:46.485297 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:46.649524 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:46.674180 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:46.677212 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:46.728420 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:46.761508 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:46.768316 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:19:46.796350 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:46.800034 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:46.834755 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:46.902357 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:47.089066 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:47.114892 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:47.166143 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:47.190922 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:47.428455 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:47.470599 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:47.506444 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:47.547688 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:47.567173 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:47.626710 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:47.652989 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:47.707795 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:47.773046 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:47.804525 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:47.821215 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:47.852890 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:47.879202 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:47.901522 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:47.902902 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:47.969362 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:47.981812 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:47.991217 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:48.007457 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:48.051221 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:48.061928 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:48.081521 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:48.113673 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:48.195848 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:48.240818 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:48.243212 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:48.311713 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:48.345700 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:48.355017 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:48.387529 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:48.406241 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:48.437150 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:48.465169 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:48.540130 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:48.540400 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:48.598842 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:48.708357 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240503 08:19:48.715188 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:48.726675 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:48.755729 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:48.809274 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:48.870465 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:49.064024 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:49.066951 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:49.097862 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:49.157519 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:49.179490 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:49.198137 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240503 08:19:49.198997 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:49.207582 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:49.256410 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:49.279242 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:19:49.317487 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:49.347957 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:49.387212 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:49.459648 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:49.460512 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:49.538596 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:49.575392 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:49.598948 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 08:19:49.626192 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240503 08:19:49.740669 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:49.930671 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:49.933231 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:50.022377 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:50.025871 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:50.042193 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:50.067538 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:50.206518 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:50.225095 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:50.234378 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:50.338081 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:50.351922 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:50.400555 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:50.410007 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:50.424680 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:50.533545 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:50.535719 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:50.539785 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:50.575251 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:50.579052 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:50.588618 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:50.601336 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:50.608384 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:50.639240 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:50.719525 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:50.742287 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:50.754700 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:50.922896 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:50.946333 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:50.957772 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:50.965481 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:50.992089 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:50.998288 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:51.026577 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:51.082393 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:51.149880 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:51.189595 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:51.343537 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:51.403880 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:51.412103 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:51.431759 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:51.447468 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:51.486493 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:51.507601 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:51.521919 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:51.596093 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:51.619942 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:51.701880 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:51.740192 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:51.743281 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:51.784711 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:51.897099 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:52.013187 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:52.022223 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:52.023830 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:19:52.036216 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:52.132217 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:52.225795 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:52.342434 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:52.535250 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:52.554821 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:52.580677 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:52.634146 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:52.655464 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:52.670008 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:52.677968 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:52.769517 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:52.793679 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:52.806938 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:52.814110 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:52.829264 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:52.851020 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:52.855610 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:52.869611 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:52.947294 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:52.991982 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:53.024803 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:53.142810 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:53.163375 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:53.166992 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:53.182673 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:53.192263 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:53.220273 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:53.267766 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:53.302067 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:53.305908 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:53.380144 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:53.440779 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:53.443053 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:53.484849 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:53.515655 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:53.638695 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:53.713482 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:53.728536 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:19:53.751423 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:53.789003 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:53.802074 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:53.817907 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:53.837152 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:53.860900 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:53.861006 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:53.875465 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:53.980794 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:54.028400 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:54.083606 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:54.123751 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:54.152809 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:54.226786 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:19:54.227653 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:54.403224 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:54.506812 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:54.517397 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:54.564899 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:19:54.601912 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:54.621747 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:54.665771 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:54.668673 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 08:19:54.669456 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:54.676820 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 08:19:54.736469 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:55.014917 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:55.052450 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:55.165956 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:55.170447 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:55.199226 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:55.205137 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:55.235354 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:55.271301 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:55.273887 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:55.333833 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:55.359076 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:55.364229 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:55.365329 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:55.384907 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:55.440632 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:55.479400 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:55.485589 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:55.495781 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:55.495939 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:55.506110 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:55.549912 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:55.556008 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:55.623195 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:55.648831 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:55.754667 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:55.823714 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:55.881178 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:55.955405 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:55.995890 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:56.025286 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:56.045395 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:56.072392 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:56.117866 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:56.128118 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:19:56.168215 7266 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:19:56.168406 7266 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:19:56.169070 7266 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:19:56.170631 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:56.170858 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:19:56.170967 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:19:56.171248 7266 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:19:56.180514 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:56.211723 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:56.264815 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:56.295109 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:56.305161 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:56.371994 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:56.402513 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:56.472712 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:56.513310 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:56.514292 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:56.568801 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:56.603641 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:56.606287 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:56.687386 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:56.703593 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:56.885998 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:56.934473 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:56.992424 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:57.003717 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:57.077821 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:57.106938 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:19:57.183562 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:57.207758 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:57.253316 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:57.505736 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:57.642249 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:57.688434 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:57.700861 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:57.706027 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:57.722337 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:57.741801 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:57.742682 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:57.756137 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:57.787026 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:57.793108 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:57.857913 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:57.868001 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:57.883903 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:57.884137 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:57.885401 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:57.911478 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:57.927120 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:57.999053 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:58.029364 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:58.103139 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:58.106570 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:58.120645 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:58.126889 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:58.127844 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:58.145932 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:58.217657 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:58.350018 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:58.375905 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:58.418251 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:58.495251 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:58.516076 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:58.606175 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:58.638255 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:58.687536 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:58.701961 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:58.714916 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:58.762975 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:19:58.766078 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:58.819763 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:58.824076 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:58.828284 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:58.890811 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:58.988690 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:59.034114 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:59.043164 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:59.101992 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:59.127934 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:59.143551 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:59.251394 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:19:59.262506 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:19:59.364661 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:59.446341 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:59.461516 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:59.497596 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:59.587785 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:59.633164 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:19:59.702412 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:19:59.730798 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 08:19:59.735030 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:19:59.792253 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:19:59.819139 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:19:59.878111 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:19:59.934136 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:00.000160 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:00.181115 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:00.188613 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:00.224133 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:00.233037 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:00.253472 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:00.262779 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:00.271541 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:00.279639 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:00.288053 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:00.289628 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:00.317806 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:00.326380 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:00.492144 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:00.493913 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:00.535444 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:00.540422 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:00.582211 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:00.658975 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:00.691092 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:00.699455 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:00.710873 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:00.725379 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:00.725724 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:00.781287 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:00.791347 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:00.888223 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:00.892793 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:00.898941 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:01.123282 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:01.171530 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:01.178320 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:01.205047 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:01.211154 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:01.235512 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:01.255242 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:01.270455 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:01.302925 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:01.339179 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:01.435437 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:01.497093 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:01.579465 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:01.722756 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:01.738765 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:01.741726 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:01.749619 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:01.752240 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:01.794044 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:01.842028 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:01.963729 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:02.014403 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:02.189543 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:02.211578 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:02.230832 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:02.238054 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:02.251964 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:20:02.319684 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:02.381923 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:02.405457 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:02.560737 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:02.597862 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:02.705801 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:02.746625 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:02.751610 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:02.798853 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:02.812474 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:02.831787 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:02.839764 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:02.859655 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:02.887555 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:02.901410 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:02.946897 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:02.954054 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:02.956383 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:02.985298 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:03.001540 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:03.043098 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:03.137477 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:03.146242 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:03.168370 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:03.222222 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:03.244975 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:03.341058 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:03.424265 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:03.458421 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:03.464704 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:03.472118 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:03.526240 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:03.580256 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:03.719965 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:03.751466 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:03.770931 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 08:20:03.792340 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:03.795269 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:03.838850 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:03.845994 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:03.890400 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:03.902935 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:03.941560 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:03.956261 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:04.109680 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:04.165665 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:04.227247 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:04.324342 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:04.343550 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:20:04.344431 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:04.360023 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:04.439535 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:04.448734 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:04.502620 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:04.523896 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:04.587339 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:04.636291 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:04.705370 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:04.713021 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:04.738989 28568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 08:20:04.804613 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:20:04.830989 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:04.884232 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:04.897720 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:20:04.919656 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:04.924957 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:04.952981 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:05.007278 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:05.059563 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:05.169519 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:05.188985 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:05.235067 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:05.275216 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:05.280189 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:05.316915 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:05.387151 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:05.416010 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:05.417526 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:05.468199 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:05.472810 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:05.477154 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:05.516680 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:05.576325 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:05.611793 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:05.624722 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:05.633345 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:05.718168 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:05.726879 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:05.732821 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:05.815632 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
I20240503 08:20:05.938298 8597 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:20:05.938508 8597 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:20:05.939296 8597 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:20:05.941058 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:05.941260 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:20:05.941373 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:20:05.941613 8597 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:20:05.965297 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:05.997211 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:06.015386 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:06.039456 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:06.093436 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:06.242480 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:06.263792 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:06.269945 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:06.337098 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:06.358130 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:06.371376 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:06.384672 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:06.397207 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:06.440191 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:06.511644 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:06.601262 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:06.694594 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:06.735787 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:06.760375 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:06.772496 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:06.819106 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:06.828428 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:06.850000 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:06.926385 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:07.122999 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:07.215287 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:07.247522 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:07.255726 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:07.277406 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:07.345038 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:07.345193 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:07.411360 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:07.416558 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:07.460081 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:07.530097 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:20:07.546628 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:07.579978 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:07.632091 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:07.657881 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:07.672988 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:07.679127 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:07.756934 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:07.770191 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:07.771715 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:07.771737 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:07.786602 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:07.871798 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:07.904062 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:07.928503 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:08.026187 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:08.029783 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:08.058563 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:08.083352 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:08.257681 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:08.287889 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:08.302351 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:08.333149 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:08.367712 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:08.508775 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:08.534123 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:08.586123 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:08.628742 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:08.639974 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:08.684459 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:08.757839 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:08.765388 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:08.777760 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 08:20:08.854941 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:08.856374 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:08.928283 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:08.956400 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:09.041188 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:09.073447 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:09.091550 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:09.136267 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:09.196835 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:09.216243 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:09.258694 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:09.296309 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:09.298511 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:09.332526 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:09.351405 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:09.385416 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:09.455456 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 08:20:09.557493 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:09.590646 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:09.608433 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:09.706575 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:09.719213 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:09.752228 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:09.772177 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 08:20:09.772985 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:09.833936 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:09.851425 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:20:09.898180 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:09.911602 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:10.023447 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:20:10.030982 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:10.034289 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:10.099174 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:10.124329 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:10.145119 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:10.225494 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:10.237438 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:10.294150 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:10.364745 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:10.365872 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:10.373217 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:10.483232 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:10.496313 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:10.497889 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:10.503860 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:10.528462 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:10.620394 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:10.717845 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:10.739660 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:10.757107 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:10.861397 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:10.874843 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:10.954984 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:11.030391 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:11.159161 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:11.165469 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:11.169893 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:11.198045 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:11.231773 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:11.332841 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:11.351280 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:11.361737 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:11.362344 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:11.379238 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:11.379222 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:11.449090 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:11.615720 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:11.626965 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:11.675143 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:11.776293 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:11.821709 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:11.840531 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:11.855446 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:11.876219 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:11.898229 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:11.911896 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:11.914460 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:11.935640 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:12.072175 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:12.072228 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:12.077252 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:12.111917 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:12.136575 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:12.164831 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:12.239531 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:12.287407 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:12.306357 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:12.414717 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:12.526554 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:12.609712 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:12.625550 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:12.638362 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:12.661495 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:12.661502 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:12.758304 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:12.773053 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:20:12.797916 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:12.852190 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:12.898677 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:12.904111 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:13.007198 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:13.025259 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:13.049875 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:13.088917 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:13.130467 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:13.137495 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:13.155155 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:13.189678 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:13.216838 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:13.471973 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:13.483211 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:13.519450 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:13.525883 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:13.646626 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:13.718082 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:13.740240 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:13.751971 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:13.752769 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:13.771407 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:13.780687 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:20:13.802309 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:13.864655 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:13.991192 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:14.050626 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:14.086850 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:14.099308 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:14.136441 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:14.198626 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:14.260792 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:14.320371 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:14.335409 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:14.407265 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:14.534204 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:14.540282 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:14.542676 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:14.542933 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:14.545560 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:14.563661 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:20:14.708787 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:14.722798 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:14.723687 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:14.744205 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:14.747265 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:14.765954 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:14.773249 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:14.782366 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:14.788750 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 08:20:14.843746 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:14.921603 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 08:20:15.051095 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:15.097615 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:15.121574 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:15.123109 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:15.125078 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:15.158785 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:15.188361 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:15.259857 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:15.292623 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:15.300655 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:15.311208 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:15.358801 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:15.412256 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:15.439554 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:15.457612 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:20:15.557730 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:15.683430 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:15.694533 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:15.707320 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:15.719169 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:15.743350 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:15.774183 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:15.822105 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:15.946125 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:16.050381 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:16.098464 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:16.137543 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:16.152065 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:16.181216 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:16.190277 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:16.205118 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:16.272413 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:16.321362 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:16.339728 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:16.415681 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:16.424250 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:16.489322 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:16.506801 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:16.717132 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:16.736461 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:16.767966 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:16.769649 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:16.794781 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:16.813742 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:16.837122 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:16.848235 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:16.916554 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:16.932016 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:17.125530 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:17.154626 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:17.189302 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:17.201879 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:17.252429 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:17.287684 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:17.344718 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:17.392961 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:17.399621 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:17.421732 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:17.503167 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:17.524102 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:17.584618 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:17.630179 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:17.634941 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:20:17.670224 28222 maintenance_manager.cc:419] P e8f5acc8744f4dccbfd654a6d7dab5bf: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033401
I20240503 08:20:17.675737 28221 maintenance_manager.cc:643] P e8f5acc8744f4dccbfd654a6d7dab5bf: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.001s sys 0.001s Metrics: {"bytes_written":2037,"delete_count":0,"lbm_write_time_us":195,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":267,"threads_started":1,"update_count":24,"wal-append.queue_time_us":391}
I20240503 08:20:17.676263 28222 maintenance_manager.cc:419] P e8f5acc8744f4dccbfd654a6d7dab5bf: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.135232
I20240503 08:20:17.688138 28221 maintenance_manager.cc:643] P e8f5acc8744f4dccbfd654a6d7dab5bf: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.012s user 0.012s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":15235,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":720,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":154,"lbm_read_time_us":289,"lbm_reads_lt_1ms":17,"lbm_write_time_us":294,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":507,"threads_started":2,"update_count":24}
W20240503 08:20:17.688827 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:17.694877 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:17.769771 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:17.835013 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:17.875793 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:17.905609 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:17.944727 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:17.986711 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:18.000418 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:20:18.060202 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:18.130988 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:18.133986 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:18.197022 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:18.256099 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:18.260833 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:18.297649 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:18.348793 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:18.354178 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:18.394062 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:18.563701 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:18.593511 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:18.646258 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:20:18.650207 9385 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:20:18.650409 9385 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:20:18.651185 9385 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:20:18.652768 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:18.652983 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:20:18.653069 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:20:18.653383 9385 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:20:18.668429 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:18.742460 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:18.769153 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:18.777124 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:18.786309 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 08:20:18.847460 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:18.870975 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:18.882884 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:18.884447 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:18.920704 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:19.122229 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:19.200237 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:19.286684 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:19.300010 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:19.324486 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:19.350670 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:19.388962 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:19.401609 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:19.448720 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:19.451506 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:19.455832 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:19.493774 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:19.545336 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:19.618217 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:20:19.724107 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:19.727211 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:19.746809 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:19.804940 28568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 08:20:19.836426 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:19.846064 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:19.852991 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:19.897364 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:19.902680 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:19.918874 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:20.023988 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:20.050257 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:20.086827 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240503 08:20:20.087663 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:20.143319 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:20.167608 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:20.195748 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:20.235286 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:20.268113 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:20.353587 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:20.374680 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:20.375735 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:20.444949 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:20.454140 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:20:20.467059 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:20.485785 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:20.518400 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:20.620272 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:20.719257 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:20.728137 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:20.751080 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:20.762555 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:20.815470 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:20.819051 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:20.953886 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:21.065624 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:21.122810 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:21.184633 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:21.217631 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:21.256978 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:21.270097 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:21.283594 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:21.290463 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:21.293848 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:21.325006 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:21.325203 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:21.428282 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:21.723359 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:21.746024 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:21.755942 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:21.777123 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:21.780247 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:21.787907 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:21.926110 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:21.946686 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:21.964702 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:21.982827 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:22.001366 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:22.015475 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:22.060931 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:22.070046 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:22.077883 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:22.201941 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:22.264569 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:22.265213 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:22.312422 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:22.340755 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:22.355937 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:22.439164 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:22.451984 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:22.589596 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:22.618285 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:22.647881 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:22.657264 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:22.717118 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:22.738776 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:22.829596 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:20:22.831820 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:22.847949 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:22.869848 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:22.949743 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:22.952425 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:22.994889 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:23.014001 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:23.046185 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:23.070127 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:23.106802 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:23.162400 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:23.171097 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:23.215849 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:23.233448 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:23.287065 28219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:20:23.288290 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:23.288285 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d60a162155504975a39c20eff88341e1: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:20:23.480291 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:23.481699 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:23.617134 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:23.663995 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:23.682111 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:23.732112 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:23.749604 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:23.776752 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:23.780185 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:23.797081 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:23.808279 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:23.866392 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:23.874464 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240503 08:20:24.114616 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:24.122709 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:24.173389 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:24.252219 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:24.259660 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:24.286685 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:24.324074 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:24.365342 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:24.376201 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:24.488821 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:24.501511 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:24.521443 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:24.555486 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:24.567368 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:24.585165 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:24.617857 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:24.647641 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:24.652758 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:24.701206 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:20:24.726831 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:24.741575 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:24.768522 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:24.807847 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:24.833330 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 08:20:24.859503 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d13c33b8712e4c488abb99b73ff42b86: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:20:24.934283 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:24.951386 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:25.024961 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:25.026362 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:25.038944 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:25.111714 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:25.120939 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 08:20:25.171097 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:25.175230 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:20:25.184454 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:25.309774 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:25.320361 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:25.366675 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:25.426815 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:25.492151 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:25.496785 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:25.513296 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:25.516849 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:25.525245 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:25.546954 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:25.556033 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:25.558144 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:25.608577 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:25.792078 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:25.889963 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:25.902623 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:25.983618 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d44335d8be7245b1bb52b2eab9e42c18: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:20:26.025528 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:26.053468 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:26.082636 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:26.105923 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:26.194842 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:26.196828 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:26.370913 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:26.377957 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:26.417467 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:26.434582 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:26.442337 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:26.443513 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:26.497555 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:26.529940 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:26.585599 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:26.748520 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:26.786626 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:26.845857 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:26.879498 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:26.938093 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:26.952780 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:26.999660 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:27.011456 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:27.062556 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:27.075026 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:27.099778 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:27.114368 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:27.157636 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:27.207329 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:27.255914 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:27.257182 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:27.274966 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:27.300136 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:27.362588 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:27.371946 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:27.379076 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:27.477202 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:27.484234 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:20:27.499819 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:27.500429 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:27.578433 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:27.627525 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:27.691978 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:27.730696 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:27.758448 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:27.845207 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:27.859226 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:27.880424 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:27.929548 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:27.941591 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:27.949968 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:27.951877 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:28.137461 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:28.167008 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:28.192047 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:28.199482 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:28.208927 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:28.222059 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:28.244683 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:28.429170 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:28.552567 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:28.598492 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:28.631043 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:28.639168 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:28.708703 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:28.718514 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:28.746372 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:28.886682 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:20:28.887588 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:28.992861 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:29.024360 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:29.028548 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:29.033856 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:29.130017 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:29.160612 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:29.171602 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:29.244062 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:29.252416 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:29.269174 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:29.336638 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:29.341507 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:29.352259 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:29.470687 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:29.481118 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:29.482574 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:29.535256 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:29.551766 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:29.563657 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:29.603899 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:29.683583 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:29.710789 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:29.715380 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:20:29.757701 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:29.763594 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:29.908838 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 08:20:29.909751 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:29.973446 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:30.059082 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:30.082249 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:30.124703 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 08:20:30.125559 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:20:30.157905 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:30.164965 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:30.196388 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:30.225817 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:30.228801 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:30.288673 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:30.291653 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:30.356828 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:30.541879 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:30.555089 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:30.578150 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:30.580761 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:30.626763 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:30.654556 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:30.710701 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:30.738139 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:30.799360 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:30.832358 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:20:30.910532 9731 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:20:30.910728 9731 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:20:30.911435 9731 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:20:30.912959 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:30.913141 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:30.913153 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
I20240503 08:20:30.913242 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:20:30.913470 9731 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:20:30.957935 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:30.975396 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:30.998170 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:31.104009 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:31.125046 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:31.134564 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:31.149966 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:31.257820 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:31.359966 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:31.412235 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:31.479902 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:31.499914 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:31.523581 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:31.582576 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:31.632393 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:31.643348 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:31.666847 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:31.686251 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:31.708156 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:31.756083 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:31.769433 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:31.851887 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:31.864188 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:31.897229 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:32.081780 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:32.082746 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:32.087003 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:32.090332 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:32.116490 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:32.119521 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:32.150534 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:32.245474 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:32.273077 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:32.341672 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:32.406953 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:32.509128 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:32.558945 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:32.590592 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:20:32.607406 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:32.626226 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:32.635375 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:32.821257 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:32.872174 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:32.883472 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:32.911290 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:32.936229 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:32.960203 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:32.998549 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:33.056125 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:33.057638 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:33.101572 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:33.139716 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:33.178082 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:33.275940 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:33.319187 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:33.472612 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:33.488905 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:33.554162 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:33.571277 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:33.577116 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:33.586215 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:33.604730 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:33.606088 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:33.608841 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:33.660300 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:33.727279 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:33.734573 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:33.795984 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:33.926097 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 08:20:34.008098 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:34.022418 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:34.085469 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:34.109773 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:34.131968 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:34.214215 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:34.219115 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:34.238853 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:34.257879 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:34.263914 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:34.287374 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:34.302836 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:34.341856 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:34.364382 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:34.471262 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:34.497817 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:34.597687 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:34.646885 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:34.710656 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:34.736498 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:20:34.741978 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:34.771396 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:34.772974 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:34.816231 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:34.883631 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:34.936530 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 08:20:35.087924 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:35.103871 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:35.127703 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:35.132315 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 08:20:35.133497 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:35.141722 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:35.153781 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:35.210017 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:35.217078 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:20:35.248987 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:35.340834 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:35.385591 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:35.409211 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:35.450503 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:35.507558 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:35.546648 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:35.614624 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:35.633519 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:35.640801 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:35.654286 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:35.711103 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:35.823973 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:35.920387 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:35.920473 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:35.924028 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:35.955996 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:35.964859 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:35.987141 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:36.012363 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:36.039398 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:36.118652 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:36.123946 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:36.146381 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:36.185701 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:36.410647 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:36.436511 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:36.516059 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:36.520079 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:36.604830 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:36.661767 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:36.677960 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:36.703091 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:36.729377 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:36.767948 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:36.805419 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:36.839080 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:36.867156 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:36.898268 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:36.911577 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:36.913806 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:36.948654 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:36.956359 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:36.969461 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:36.971709 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:36.975061 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:37.059085 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:37.146976 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:37.225407 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:37.374624 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:37.391968 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:37.425442 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:37.505208 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:37.563863 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:37.611258 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:37.669314 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:37.687781 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:37.814925 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:20:37.825471 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:37.884191 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:37.901496 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:37.929162 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:37.958134 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:37.987066 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:38.005393 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:38.030921 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:38.073458 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:38.078576 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:38.122097 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:38.154270 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:38.186115 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:38.350687 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:38.411768 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:38.445119 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:38.481242 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:38.543375 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:38.549821 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:38.587126 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:38.597533 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:38.648626 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:38.694376 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:38.790745 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:38.791838 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:38.867244 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:38.912184 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:38.952308 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:20:39.025462 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:39.040748 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:39.091686 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:39.165652 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:39.240525 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:39.252099 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:39.255846 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:39.266324 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:39.291302 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:39.324946 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:39.334388 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:39.361326 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:39.378641 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:39.380123 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:39.383251 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:39.387008 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:39.433681 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:39.590261 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:39.634855 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:39.694413 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:39.791253 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:20:39.792029 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:39.823406 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:39.827253 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:39.929611 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:39.930475 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:39.932586 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:40.012602 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:40.015450 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 08:20:40.048938 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:40.210783 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:40.216109 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:40.226087 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:40.229657 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:40.243252 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:20:40.254195 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:40.336762 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:20:40.390309 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:40.417869 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:40.473292 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:40.498096 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:40.499300 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:40.510414 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:40.564363 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:40.602150 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:40.727409 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:40.738190 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:40.778776 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:40.825845 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:40.856235 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:40.970479 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:40.996395 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:41.016464 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:41.022125 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:41.151600 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:41.180088 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:41.238816 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:41.302453 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:41.312958 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:41.332590 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:41.370024 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:41.458194 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:41.512600 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:41.522632 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:41.565873 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:41.566074 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:41.637140 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:41.677582 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:41.775370 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:41.782716 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:41.808624 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:41.810642 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:41.820349 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:41.821043 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:41.922545 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:42.001415 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:42.016930 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:42.027070 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:42.077772 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:42.135933 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:42.152109 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:42.240370 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:42.259255 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:42.330039 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:42.395656 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:42.492166 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:42.554261 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:42.567088 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:42.615270 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:42.626812 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:42.636708 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:42.714736 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:42.753188 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:42.757827 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:20:42.782979 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:42.923739 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:42.935608 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:42.946472 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:42.969895 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:42.972890 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:42.978286 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:43.026698 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:43.109427 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:43.146274 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:43.175603 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:43.246552 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:43.316946 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:43.345096 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:43.376842 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:43.560436 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:43.579914 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:43.585358 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:43.666105 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:43.734457 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:43.740806 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:43.767462 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:43.768435 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:43.799386 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:43.880582 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:43.963358 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:43.988248 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:20:44.001863 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:44.016058 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:44.038513 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:44.042464 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:44.047926 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:44.049788 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:44.059811 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:44.073072 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:44.207731 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:44.255093 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:44.337484 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:44.396944 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:44.439143 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:44.469341 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:44.524849 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:44.528982 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:44.610018 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:44.647348 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:44.675094 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:44.698640 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:44.703848 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:44.790903 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:44.829730 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 08:20:44.830646 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:44.880113 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:44.984609 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:44.990810 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:45.056902 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:45.144181 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:45.151960 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 08:20:45.161283 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:45.178221 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:45.199386 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:20:45.247066 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:45.259009 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:20:45.259897 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:45.333060 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:45.394382 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:45.400696 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:45.495920 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:45.568198 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:45.573063 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:45.575806 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:45.626176 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:45.642386 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:45.850844 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:45.966639 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:45.973522 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:45.989866 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:46.007529 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:46.015580 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:46.026228 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:46.079713 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:46.129307 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:46.159426 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:46.161310 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:46.195258 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:46.195322 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:46.327864 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:46.331418 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:46.380829 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:46.387077 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:46.423008 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:46.455736 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:46.476336 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:46.487056 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:46.520431 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:46.620635 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:20:46.725571 11256 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:20:46.725775 11256 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:20:46.726426 11256 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:20:46.728191 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:46.728377 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:20:46.728459 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:20:46.728696 11256 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:20:46.744442 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:46.750735 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:46.785207 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:46.957281 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:46.968202 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:47.009987 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:47.051304 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:47.109858 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:47.119871 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:47.123437 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:47.142848 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:47.149691 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:47.247771 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:47.258276 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:47.273685 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:47.279412 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:47.405313 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:47.427456 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:47.431612 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:47.559049 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:47.592787 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:47.617076 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:47.642194 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:47.648322 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:20:47.709017 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:47.832090 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:47.861121 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:47.929803 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:47.957849 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:47.986680 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:48.003021 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:48.004233 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:48.010000 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:48.127139 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:48.182011 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:48.368834 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:48.389991 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:48.397874 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:48.422106 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:48.452051 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:48.504494 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:48.530772 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:48.536226 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:48.571238 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:48.580813 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:48.612118 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:48.693987 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:48.741357 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:48.749056 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:48.760318 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:48.820657 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:48.824097 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:48.906091 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:48.935029 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:48.988468 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:48.993227 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:20:49.013870 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:49.030604 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:49.153373 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:49.290864 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:49.350679 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:49.350803 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:49.403566 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:49.425778 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:49.427174 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:49.533764 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:49.534462 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:49.574214 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:49.574373 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:49.590145 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:49.593400 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:49.684924 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:49.695299 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:49.713398 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:49.838726 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:20:49.839738 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:49.873605 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:49.932647 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:49.952575 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:49.955883 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:50.006182 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:50.143430 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:50.160796 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 08:20:50.219368 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:50.321269 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:20:50.322178 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:20:50.391297 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:50.414960 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:50.429745 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:50.446606 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:50.509089 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:50.511870 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:50.518383 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:50.522990 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:50.571916 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:50.628536 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:50.781968 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:50.840396 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:50.860831 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:50.976583 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:50.977566 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:50.996706 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:51.001335 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:51.038146 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:51.038230 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:51.065994 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:51.106950 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:51.118850 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:51.147361 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:51.165019 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:51.267284 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:51.316177 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:51.340180 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:51.399624 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:51.513043 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:51.572759 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:51.609181 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:51.626504 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:51.729811 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:51.790586 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:51.899910 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:51.901803 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:51.921867 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:51.950289 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:51.964267 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:51.966348 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:51.997210 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:52.082895 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:52.092854 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:52.130764 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:52.183830 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:52.213438 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:52.234764 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:52.236518 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:52.270845 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:52.319816 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:52.445750 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:52.518481 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:52.552413 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:52.607268 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:52.615584 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:52.748761 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:52.749053 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:20:52.823944 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:52.869925 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:52.913417 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:52.925778 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:52.997956 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:53.011068 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:53.036723 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:53.051175 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:53.130054 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:53.173319 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:53.282907 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:53.298063 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:53.307597 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:53.322747 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:53.411695 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:53.422200 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:53.451926 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:53.474146 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:53.541162 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:53.558001 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:53.568684 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:53.608301 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:53.618474 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:53.855877 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:53.862881 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:53.897480 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:53.948168 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:53.986800 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:53.993500 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:54.022922 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 08:20:54.023855 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:54.093160 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:54.132537 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:54.160526 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:54.245602 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:54.326632 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:54.327174 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:54.329986 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:54.474133 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:54.490859 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:54.579571 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:54.585975 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:54.626793 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:54.647359 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:54.683862 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:54.724512 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:54.775918 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:54.839111 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240503 08:20:54.840181 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:54.869891 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:54.877470 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:54.896090 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:54.914557 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:55.010565 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:55.041813 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:55.067045 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:55.071607 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:55.089288 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:55.093736 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:55.234501 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:55.259822 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 08:20:55.300922 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:20:55.309605 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:55.326035 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 08:20:55.393755 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:55.408298 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:55.420473 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:55.456665 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:55.461560 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:55.469347 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:55.611244 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:55.647794 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:55.649535 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:55.712899 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:55.747296 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:55.805910 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:55.866050 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:55.950323 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:55.969024 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:56.005205 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:56.009807 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:56.018937 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:56.054097 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:56.058966 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:56.173514 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:56.203524 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:56.215461 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:56.460827 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:56.480020 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:56.510418 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:56.537961 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:56.581118 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:56.640197 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:56.664945 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:56.715869 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:56.718303 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:56.747809 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:56.763738 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:56.771572 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:56.879206 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:56.983673 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:57.062970 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:57.073197 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:57.142853 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:57.190021 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:57.195246 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:57.200968 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:57.258107 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:57.343602 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:57.358203 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:57.385246 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:57.390203 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:57.428246 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:57.441229 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:57.451628 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:57.456777 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:57.476473 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:57.526762 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:57.644469 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:57.656335 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:57.664670 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:20:57.697767 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:57.862967 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:57.942641 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:57.956602 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:57.964972 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:57.967957 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:57.980578 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:58.118381 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:58.174690 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:58.193753 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:58.199471 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:20:58.274735 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:58.290180 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:58.310204 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:58.356020 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:58.369199 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:58.421005 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:58.445586 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:58.558485 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:58.583047 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:58.645815 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:58.694375 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:58.791021 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:58.798106 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:58.804807 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:58.814846 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:58.997184 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:59.114356 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:59.137799 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 08:20:59.138726 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:59.149112 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:59.247258 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:59.281767 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:59.287807 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:20:59.291389 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:59.376557 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:59.399288 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:59.409293 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:59.410342 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:59.429468 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:59.502542 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
I20240503 08:20:59.550860 12216 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:20:59.551039 12216 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:20:59.551719 12216 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:20:59.553341 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:59.553508 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:20:59.553617 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:20:59.553825 12216 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:20:59.713529 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:59.726876 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:59.733311 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:20:59.784816 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:59.785179 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:59.789093 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:20:59.806164 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:20:59.807757 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:59.827999 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:20:59.843261 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:59.861189 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:20:59.894382 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:20:59.899130 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 08:20:59.956634 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:20:59.976629 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:20:59.984055 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:21:00.104959 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:21:00.123894 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:21:00.177405 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:21:00.254805 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:00.261953 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 08:21:00.262826 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:00.285523 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:21:00.329205 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:21:00.409929 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:00.412261 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:00.541862 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:00.664942 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:21:00.693773 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:00.718575 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:00.738466 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:21:00.814694 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:21:00.823599 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:00.853556 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:00.951570 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:01.006496 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:01.008406 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:01.033556 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:21:01.038470 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:21:01.165030 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:01.179703 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:21:01.201465 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:01.284431 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:01.379001 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:01.409587 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:01.411948 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:01.497776 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:01.518574 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:01.554983 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:01.569768 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:01.621392 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:01.640551 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:01.706782 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:01.838076 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:21:01.855825 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:01.913702 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:01.938638 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:01.975524 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:02.034303 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:02.089927 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:02.122867 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:02.123005 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:02.130550 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:02.156684 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:02.214941 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:21:02.236950 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:02.251983 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:02.256474 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:02.262046 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:02.280938 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:21:02.410629 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:02.419677 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:02.424571 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:02.444545 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:21:02.561033 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:02.571564 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:02.632464 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:02.643229 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:21:02.676412 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:02.773445 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:02.805194 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:02.846338 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:02.920501 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:03.118845 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:03.156911 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:03.159071 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:03.185591 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:03.196679 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:03.223042 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:03.262770 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:03.317612 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:03.329461 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:03.439860 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:03.457165 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:03.532572 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:03.655805 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:03.696173 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:03.714018 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:03.715315 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:03.743077 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:03.768222 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:03.773109 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:03.816372 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:03.840852 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:03.959977 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:04.079061 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:04.117735 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:04.125771 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:04.146801 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:04.178159 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:21:04.184830 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:04.216563 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:04.227401 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:04.241190 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:04.407145 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:04.421864 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:04.491493 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:04.544932 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:04.594244 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:04.641803 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:04.663429 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:04.667251 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:04.669683 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:04.679508 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:04.693792 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:04.700980 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:04.762565 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:04.790449 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:04.824990 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:04.855155 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:04.899180 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:04.982482 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:21:04.983378 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:05.118402 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:05.167759 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:05.168450 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:05.184733 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:05.236217 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:21:05.263715 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 08:21:05.325420 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:05.332007 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:05.353389 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:21:05.461971 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:05.474673 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:05.494086 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:05.580991 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:05.609551 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:05.651537 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:05.690510 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:05.698823 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:05.718765 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:05.798460 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:05.805770 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:05.812536 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:05.828107 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:05.921394 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:05.941365 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:05.969666 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:05.988075 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:06.117662 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:06.200104 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:06.228849 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:06.251379 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:06.352357 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:06.355665 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:06.417280 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:06.451587 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:06.488994 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:06.673954 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:06.695291 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:06.711609 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:06.767081 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:06.807940 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:06.809473 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:06.836673 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:06.843412 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:06.862198 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:06.869771 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:06.886758 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:06.943229 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:06.956539 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:06.964496 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:07.028818 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:07.090916 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:07.103156 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:07.227720 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:07.313513 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:07.318037 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:07.319648 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:07.336638 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:07.364182 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:07.380564 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:07.400353 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:07.557255 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:07.563387 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:07.633431 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:07.644812 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:21:07.764081 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:07.783579 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:07.794551 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:07.808326 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:07.830741 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:07.911629 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:07.927677 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:07.936726 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:08.053516 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:08.099574 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:08.118753 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:08.187510 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:08.259900 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:08.283231 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:08.304553 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:08.330973 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:08.477933 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:08.480585 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:08.540144 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:08.615456 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:08.698336 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:08.707772 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:08.781772 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:08.784449 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:08.824724 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:08.868930 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:08.883014 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:08.929744 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:09.044147 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:09.059573 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:09.103132 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:09.106125 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:09.192145 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:21:09.193053 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:09.301909 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:09.317358 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:09.345934 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:09.363364 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:09.389672 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:09.390626 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:09.392382 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:09.400544 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:09.446631 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:09.539108 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:09.568928 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:09.584324 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:09.599000 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:09.650895 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:09.658811 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:09.723462 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:09.741736 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:09.815614 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:09.818601 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:09.948966 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:09.975586 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:09.997475 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:21:10.049758 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:10.057884 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:10.074949 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:10.078966 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:10.109094 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:10.239990 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:10.255510 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:21:10.261978 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:10.267802 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 114 similar messages]
W20240503 08:21:10.332144 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:10.397864 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:10.413228 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 08:21:10.486908 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:10.520344 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:10.657649 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:10.671082 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:10.681123 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:10.718611 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:10.771328 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:10.781476 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:10.803814 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:10.865406 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:10.880859 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:10.919410 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:10.988293 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:11.080556 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:11.110745 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:11.206112 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:11.279891 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:11.339087 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:11.389416 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:11.389695 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:11.423126 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:11.435514 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:11.448627 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:11.523409 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:11.694208 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:11.698521 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:11.716706 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:11.767791 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:11.787007 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:11.850337 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:11.855996 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:11.915978 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:11.920521 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:11.921063 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:11.947993 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:11.979142 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:12.081738 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:12.094799 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:12.107321 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:12.121094 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:12.125396 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:12.147614 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:12.262646 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:12.311790 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:12.339051 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:12.390120 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:12.397320 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:12.410584 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:12.424891 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:12.435959 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:12.524164 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:12.638511 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:12.725919 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:12.801992 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:12.881887 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:12.886423 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:12.890568 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:12.904369 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:12.934020 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:21:12.949616 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:13.016366 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:13.106389 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:13.141161 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:13.260090 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:13.303288 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:13.390887 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:13.394134 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:13.422312 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:13.439635 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:13.478710 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:13.489101 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:13.537670 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:13.688736 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:13.790208 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:13.818238 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:13.833405 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:13.867460 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:13.905674 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:13.982426 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:13.990079 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:13.993244 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:13.994729 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:13.994946 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:14.005175 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:14.058316 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:14.203064 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:14.205291 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:21:14.267482 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:14.342063 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:14.366816 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:14.405169 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:14.408833 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:14.467365 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:14.476336 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:14.495609 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:14.515983 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:14.529145 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:14.572994 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:14.611901 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:14.722362 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:14.723501 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:14.787680 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:14.832360 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:14.885475 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:14.899283 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:14.938239 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:14.993749 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:15.016252 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 08:21:15.017093 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:15.029780 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:15.074575 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:15.126938 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:15.139581 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:15.219421 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:15.232096 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:15.279322 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:15.306289 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:15.312402 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 08:21:15.371940 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:21:15.461457 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:15.530974 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:21:15.584556 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:15.590446 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:15.590508 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:15.668738 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:15.735633 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:15.747500 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:15.748145 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:15.767118 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:15.778709 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:15.893771 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:15.929394 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:16.108646 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:16.133821 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:16.223913 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:16.307569 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:16.311628 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:16.312934 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:16.363447 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:16.367823 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:16.372699 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:16.389781 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:16.415850 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:16.540953 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:16.562260 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:16.576193 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:16.586552 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:16.622922 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:16.648932 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:16.732317 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:16.804744 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:16.858471 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:16.916783 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:16.917675 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:17.022156 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:17.045768 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:17.055161 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:17.064507 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:17.091706 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:17.170809 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:17.243022 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:17.276108 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:17.282970 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:17.306056 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:17.309154 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:17.347486 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:17.393441 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:17.456895 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:17.485173 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:17.510018 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:21:17.627069 13270 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:21:17.627274 13270 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:21:17.628002 13270 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:21:17.629930 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:17.630364 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:21:17.630470 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:21:17.630717 13270 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:21:17.670159 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:17.776937 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:17.791131 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:17.834206 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:17.837188 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:17.853938 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:17.861205 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:17.863025 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:21:17.888495 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:17.889039 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:17.901315 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:17.924777 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:17.988283 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:18.032553 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:18.118016 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:18.264838 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:18.268913 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:18.354277 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:18.389780 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:18.432022 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:18.487015 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:18.530705 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:18.780009 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:18.864207 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:18.904680 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:18.922264 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:18.922312 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:18.931740 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:18.937884 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:18.959998 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:18.966915 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:18.983961 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:19.049568 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:19.076431 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:19.213302 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240503 08:21:19.237360 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:19.288523 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:19.348101 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:19.359324 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:19.394433 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:19.460786 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:19.563939 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:19.596240 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:19.605912 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:19.638458 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:19.665732 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:19.685393 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:19.774195 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:19.790673 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:19.795867 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:19.804786 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:19.806077 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:19.897064 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:19.952365 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:19.957154 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:19.958464 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:20.027577 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:21:20.028611 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:20.049269 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:20.173008 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:20.207841 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:20.221117 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:20.224275 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:20.267120 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:20.322603 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:20.359421 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 08:21:20.363978 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:20.375669 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:20.390194 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:20.400430 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:20.428337 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:20.449623 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:20.471678 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:20.538040 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:20.545814 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:20.565999 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:21:20.669963 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:21:20.799966 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:20.802886 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:20.807718 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:20.848019 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:20.903513 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:21.119757 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:21.199092 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:21.252776 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:21.254688 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:21.302210 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:21.363965 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:21.448328 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:21.459591 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:21.538569 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:21.543079 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:21.600589 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:21.620599 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:21.678462 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:21.816342 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:21.820967 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:21.836750 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:21.838682 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:21.850932 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:21.956424 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:21.957509 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.059856 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:22.110612 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.139705 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.146113 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.186820 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:22.209399 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:22.212570 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.251734 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.282016 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.335845 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:22.344774 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:22.361507 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:22.453794 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:22.492051 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.549638 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.556396 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.574553 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:22.585997 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.660162 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.691340 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.736079 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.748937 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.754266 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:22.780858 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:22.847771 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:22.867103 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:22.880194 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:22.899083 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:22.947554 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:22.979110 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:22.986990 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:23.068122 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:23.187326 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:23.279816 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:23.293205 28219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:21:23.294339 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d60a162155504975a39c20eff88341e1: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:21:23.336213 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:23.351701 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:21:23.423326 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:23.557477 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:23.565268 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:23.631721 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:23.673341 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:23.676265 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:23.711987 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:23.953187 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:24.041910 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:24.080020 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:24.098292 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:24.109303 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:24.115998 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:24.176617 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:24.207567 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:24.257531 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:21:24.297678 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:24.301923 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:24.343542 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:24.452970 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:24.479923 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:24.536034 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:24.583784 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:24.592993 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:24.622562 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:24.639518 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:24.660302 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:24.663574 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:24.687925 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:24.703680 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:24.707473 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:24.776412 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:24.866103 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:24.880831 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d13c33b8712e4c488abb99b73ff42b86: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:21:24.922021 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:24.963861 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:24.966055 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:24.995983 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:25.036482 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 08:21:25.037451 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:25.052067 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:25.096647 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:25.114972 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:25.154503 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:25.155354 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:25.175818 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:25.181710 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:25.183420 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:25.215991 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:25.274577 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:25.328716 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:25.363054 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 08:21:25.366899 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:25.435755 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:25.488755 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:25.551314 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:25.566946 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:25.588132 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 08:21:25.606446 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:25.630105 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:25.644148 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:25.665526 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:25.924468 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:21:26.017788 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d44335d8be7245b1bb52b2eab9e42c18: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:21:26.019773 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:26.050118 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:26.092787 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:26.136679 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:26.155669 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:26.202870 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:26.204115 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:26.213879 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:26.455487 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:26.464598 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:26.467662 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:26.565217 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:26.586225 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:26.642119 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:26.668246 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:26.766636 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:26.795249 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:26.901520 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:26.909300 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:26.949360 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:27.023151 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:27.044256 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:27.047386 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:27.096223 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:27.130477 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:27.146970 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:27.191016 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:27.217959 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:27.226960 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:27.244634 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:27.300091 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:27.328619 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:27.329257 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:27.398582 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:27.398716 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:27.431829 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:27.436156 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:27.476845 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:27.479131 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:27.555371 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:27.556617 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:27.654618 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:27.654892 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:27.671976 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:27.691757 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:27.766917 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:27.822311 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:27.863132 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:27.865414 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:27.965646 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:28.020246 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:28.024614 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:28.206460 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:28.221611 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:28.240119 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:28.365236 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:28.373724 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:21:28.374027 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:28.413182 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:28.535598 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:28.625396 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:28.698143 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:28.720660 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:28.729266 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:28.745586 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:28.824299 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:28.824337 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:28.868018 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:28.954007 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:29.113829 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:29.115099 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:29.163898 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:29.279717 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240503 08:21:29.316006 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:29.332262 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:29.347163 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:29.348436 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:29.407419 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:29.455663 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:29.506695 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:29.528852 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:29.595338 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:29.598268 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:29.705268 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:29.721216 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:29.726439 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:29.729650 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:29.785323 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:29.806821 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:29.810232 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:29.816967 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:29.827080 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:29.833549 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:29.925777 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:29.939966 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:29.950557 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:29.968065 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:29.994783 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:30.004935 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:30.013556 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:30.033741 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:30.157650 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:30.206416 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:30.217506 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240503 08:21:30.218400 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:30.260768 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:30.264488 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:30.274058 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:30.322741 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:30.368894 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:30.375559 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 08:21:30.442209 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:30.592832 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:30.604236 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:21:30.669991 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:30.686678 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:30.743806 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:30.750160 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:30.812041 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:30.822297 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:30.851892 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:30.929781 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:21:31.039813 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:31.173069 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:31.213721 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:31.265388 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:31.266194 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:31.322886 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:31.330449 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:31.392021 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:31.499579 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:31.504731 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:31.571771 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:31.585471 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:31.658888 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:31.660529 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:31.676426 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:31.730911 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:31.897429 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:31.942406 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:32.007493 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:32.055956 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:32.062048 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:32.074491 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:32.114415 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:32.118528 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:32.130347 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:32.254979 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:32.265743 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:32.266309 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:32.281905 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:32.298472 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:32.298689 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:32.298714 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:32.348210 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:32.359517 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:32.373457 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:32.397289 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:32.448000 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:32.542124 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:32.547859 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:32.567317 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:32.573310 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:32.577702 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:21:32.580597 14529 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:21:32.580812 14529 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:21:32.581512 14529 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:21:32.582795 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:32.582942 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:21:32.583038 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:21:32.583345 14529 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:21:32.618127 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:32.644248 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:32.739292 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:32.768157 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:32.840591 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:32.946992 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:32.961135 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:33.089836 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:33.116568 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:33.158121 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:33.192551 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:33.224244 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:33.331663 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:33.352306 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:21:33.390553 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:33.409991 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:33.626986 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:33.630147 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:33.685279 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:33.686261 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:33.709964 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:33.738785 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:33.835964 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:33.908182 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.020085 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.032660 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:34.061810 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:34.061945 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:34.084508 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:34.180805 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.231029 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:34.301141 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:34.357673 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:21:34.417554 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.448145 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:34.538511 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:34.542812 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.543288 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:34.556175 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.590095 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.616855 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:34.622396 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.662616 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.671272 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:34.690791 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.700523 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:34.758229 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.769775 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.799185 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.803404 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.821413 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:34.848974 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:34.904371 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:34.919212 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.964659 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:34.966606 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:34.976501 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:35.052482 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:35.053992 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:35.064489 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:35.132171 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:35.231575 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:35.241721 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:21:35.242640 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:35.372963 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:35.373407 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:35.414131 28568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 08:21:35.414836 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:35.474566 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:35.526013 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:35.527472 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:35.669483 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240503 08:21:35.674036 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:35.687338 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:35.801576 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:35.802177 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:21:35.833922 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:35.912845 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:35.913060 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:36.006435 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:36.033545 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:36.038812 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:36.186221 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:36.220372 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:36.227525 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:36.238413 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:36.251451 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:36.464021 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:36.497697 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:36.535588 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:36.575984 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:36.643739 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:36.644181 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:36.741329 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:36.748351 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:36.857669 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:36.879174 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:36.898169 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:36.910449 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:36.944973 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:36.974656 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:37.047359 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:37.071867 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:37.147812 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:37.160229 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:37.247282 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:37.247440 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:37.267700 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:37.283877 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:37.303802 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:37.318964 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:37.351027 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:37.376988 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:37.377835 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:37.397272 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:37.424605 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:37.426510 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:37.444576 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:37.506217 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:37.598672 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:37.600998 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:37.605824 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:37.630896 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:37.775897 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:37.826889 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:37.846446 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:37.847173 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:37.886816 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:37.905005 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:37.981554 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:38.188340 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:38.243091 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:21:38.306176 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:38.309028 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:38.316259 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:38.319593 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:38.352202 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:38.371753 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:38.426962 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:38.570350 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:38.665212 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:38.752835 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:38.796809 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:38.799986 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:38.880738 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:38.897065 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:38.934289 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:38.934340 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:39.009514 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:39.041556 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:39.089830 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:39.170590 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:39.185351 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:39.291474 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:39.316352 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:39.316646 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:39.380163 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:39.388193 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:39.394151 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:39.397002 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:21:39.422006 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:39.443992 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:39.474923 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:39.524035 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:39.572991 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:39.618012 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:39.650009 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:39.661319 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:39.688849 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:39.694908 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:39.823562 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:39.842111 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:39.880998 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:39.903453 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:39.909703 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:39.910166 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:39.938688 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:39.964543 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:40.062736 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:40.069391 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:40.197113 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:40.269409 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:21:40.271158 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:40.275072 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:40.381901 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:40.396039 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:40.395998 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:40.447829 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:40.456106 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:21:40.457702 28568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 115 similar messages]
W20240503 08:21:40.458498 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:40.467113 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:40.553953 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:40.650743 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:40.756043 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:40.812156 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:40.818409 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:21:40.844521 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:40.855362 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:40.858492 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:41.020669 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:41.020782 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:41.146353 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:41.221169 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:41.269527 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:41.431449 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:41.488842 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:41.492839 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:41.595743 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:41.631126 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:41.635741 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:41.646229 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:41.649665 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:41.680349 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:41.692641 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:41.693070 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:41.701619 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:41.751590 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:41.777293 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:41.792790 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:41.810858 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:41.867574 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:41.878876 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:41.907182 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:41.913800 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:42.012679 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:42.080492 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:42.148484 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:42.218809 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:42.247200 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:42.298151 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:42.382732 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:42.386993 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:42.404582 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:42.490537 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:42.531984 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:42.553596 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:42.594801 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:42.657338 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:42.665179 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:42.671715 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:42.706254 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:42.717895 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:42.724128 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:42.738608 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:42.802616 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:42.855924 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:42.893285 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:42.912784 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:21:42.921036 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:42.971547 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:43.053081 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:43.085471 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:43.112581 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:43.181793 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:43.208889 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:43.296717 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:43.445824 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:43.448091 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:43.588835 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:21:43.709810 14768 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:21:43.710114 14768 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
W20240503 08:21:43.710871 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
I20240503 08:21:43.711200 14768 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:21:43.712966 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:43.713152 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:21:43.713250 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:21:43.713526 14768 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:21:43.729564 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:43.901700 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:43.942992 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:43.964773 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:43.967828 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.127357 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.128445 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.141631 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.153786 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:44.179373 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:44.216778 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.223917 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.232973 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:44.258107 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:44.279345 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:44.291523 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:44.314213 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.356822 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.402585 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:21:44.427042 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.438334 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.527079 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.549584 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:44.594758 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:44.595546 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.650523 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:44.719211 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:44.720105 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.726300 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:44.871763 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.883258 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:44.944087 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.946496 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.968663 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:44.999919 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:45.075547 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:45.095379 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:45.134600 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:45.170715 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:45.191150 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:45.242219 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:45.275545 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:21:45.276729 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:45.355419 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:45.363598 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:45.377256 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:45.378240 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:45.478452 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 08:21:45.502300 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:45.626323 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:45.632515 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:45.676174 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:45.685458 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:45.713140 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:21:45.723459 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:45.777827 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:45.821341 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:21:45.861528 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:45.907579 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:45.936611 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:46.099548 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:46.161701 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:46.234124 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:46.321517 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:46.504501 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:46.508195 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:46.549597 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:46.568415 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:46.569792 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:46.582448 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:46.594525 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:46.620477 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:46.623064 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:46.669245 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:46.687703 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:46.741780 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:46.770306 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:46.786979 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:46.793591 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:46.830220 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:46.884040 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:46.953779 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:46.978349 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:46.993108 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:47.004508 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:47.070732 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:47.169493 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:47.170042 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:47.181357 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:47.189472 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:47.224732 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:47.245111 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:47.257922 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:47.269670 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:47.283125 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:47.447576 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:47.459283 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:47.479439 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:47.575209 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:47.617758 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:47.647866 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:47.656697 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:47.803236 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:47.811663 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:47.845162 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:47.932298 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:47.993345 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:48.003293 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:48.016358 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:48.018096 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:48.104036 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:48.205498 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:48.346365 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:48.347545 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:48.351213 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:48.369210 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:21:48.382450 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:48.427058 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:48.526898 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:48.601941 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:48.605516 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:48.940622 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:48.985556 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:48.995601 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.097375 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:49.101039 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:49.130514 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.155962 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:49.164386 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.169631 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.203574 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:49.212592 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.226262 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.228920 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.285050 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:49.319540 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.327081 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:49.331506 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.356524 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.379552 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:49.406086 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:49.413702 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:21:49.471258 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.530881 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:49.530881 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.555750 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:49.557924 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:49.624349 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.663300 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.676823 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.732590 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.755986 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:49.844475 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:49.875155 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:49.876375 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:50.005344 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:50.043499 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:50.097301 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:50.134681 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:50.170747 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:50.257136 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:50.262971 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:50.276273 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:50.335513 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:50.367012 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:21:50.391323 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:50.424201 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:50.430162 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:50.457348 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:50.522406 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:50.536161 28568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 08:21:50.537087 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:50.570128 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:50.669871 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:50.679975 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:50.835764 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:50.880565 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:50.928098 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 08:21:50.953014 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:51.030395 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:51.042539 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:51.051887 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:51.086428 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:21:51.331977 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:51.425519 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:51.439153 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:51.440742 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:51.530906 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:51.544987 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:51.566390 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:51.582989 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:51.588497 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:51.615181 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:51.635803 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:51.654860 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:51.696049 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:51.738658 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:51.749449 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:51.788573 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:51.805660 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:51.846228 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:51.853499 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:51.860085 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:51.965706 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:52.004307 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:52.020401 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:52.035243 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:52.122938 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:52.202378 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:52.252502 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:52.291620 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:52.291787 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:52.357085 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:52.414005 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:52.416304 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:52.420377 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:52.429462 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:52.477183 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:52.622534 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:52.681514 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:52.702118 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:52.718770 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:52.756803 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:52.776145 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:52.809998 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:52.861790 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:52.900300 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:52.924367 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:52.951351 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:52.954322 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:53.020633 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:53.062932 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:53.127609 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:53.150242 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:53.254606 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:53.326634 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:53.347823 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:53.439045 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:21:53.444767 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:53.570739 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:53.707111 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:53.791505 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:53.813721 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:53.881376 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:53.904776 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:53.909653 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:53.934809 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:53.936851 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:53.944995 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:53.970548 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:54.041843 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:54.066315 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:54.179488 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:54.206101 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:54.246462 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:54.267781 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:54.291397 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:54.310163 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:54.345575 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:54.347626 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:54.384240 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:54.391538 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:54.420682 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:21:54.435606 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:54.467054 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:54.469310 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:54.483100 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:54.664351 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:54.708518 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:54.742058 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:54.770287 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:54.784958 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:54.789449 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:54.798239 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:54.904714 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:54.909677 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:54.913836 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:55.089350 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:55.181558 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:55.188944 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:55.207504 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:55.213166 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:55.240000 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:55.303772 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:55.319026 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:55.374313 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 08:21:55.394984 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:55.401937 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:55.406757 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:55.434298 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:55.440646 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:55.452308 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:55.507838 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:55.615669 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 115 similar messages]
W20240503 08:21:55.662647 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:55.700579 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
I20240503 08:21:55.920467 15684 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
W20240503 08:21:55.920562 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
I20240503 08:21:55.920652 15684 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:21:55.921286 15684 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:21:55.923236 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:55.923395 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:21:55.923491 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:21:55.923679 15684 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:21:55.942329 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:21:56.029122 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.033460 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:21:56.045372 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:56.107750 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.144027 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.191788 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.270228 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:56.273072 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:56.324257 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:56.375196 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.396133 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:56.452755 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.461962 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.488458 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.555070 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:56.573925 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.627629 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:56.636128 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:56.650352 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.717635 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.762763 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:56.775218 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:56.795285 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:56.817227 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.829106 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.838115 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.849714 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.963244 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:56.989589 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:57.074481 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:57.103657 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:57.155457 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:57.183108 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:57.278383 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:57.284489 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:57.323144 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:57.349390 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:21:57.389427 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:57.423468 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:57.455217 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:57.497241 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:57.567931 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:57.638073 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:57.647389 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:21:57.652958 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:57.664986 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:57.780735 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:57.787360 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:57.798347 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:57.829520 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:57.832149 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:57.889362 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:57.907202 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:57.966101 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:57.982208 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:58.102092 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:58.225346 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:58.291399 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:58.439324 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:58.442288 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:21:58.591861 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:58.593475 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:58.596098 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:21:58.633056 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:58.687861 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:58.699062 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:58.837725 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:58.887620 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:58.889694 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:58.902374 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:58.943295 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:58.993278 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:59.002007 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:59.011138 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:59.079671 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:59.082095 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:59.144754 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:59.149139 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:59.150310 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:59.209353 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:59.234369 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:59.252956 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:59.284727 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:59.367190 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:59.426573 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:59.432358 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 08:21:59.445699 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:59.563849 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:21:59.574266 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:21:59.597503 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:21:59.607440 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:21:59.644852 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:59.645650 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:59.742291 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:21:59.845463 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:21:59.866451 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:59.869678 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:21:59.897945 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:21:59.959663 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:00.054220 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:22:00.099833 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:22:00.153107 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:00.177389 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:00.184721 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:22:00.200124 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:00.219921 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:22:00.233076 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:22:00.266654 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:00.285652 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:00.342183 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:00.396054 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:22:00.404366 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:00.410312 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:22:00.434811 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 08:22:00.570778 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:00.622921 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:00.637580 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:00.679126 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 08:22:00.769881 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:00.979918 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240503 08:22:01.037884 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:01.047267 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:01.051596 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:01.082983 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:01.086503 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:01.090759 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:01.107694 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:01.179066 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:01.202024 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:22:01.243956 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:01.328049 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:01.400552 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:01.421571 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:01.470830 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:01.503926 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:22:01.558216 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:22:01.570825 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:01.600070 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:01.612264 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:01.658692 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:01.694896 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:22:01.739753 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:01.786192 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:01.849203 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:01.872481 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:01.930357 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:01.982086 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:01.990185 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:02.000692 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:02.018234 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:02.027993 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:02.095404 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:02.205693 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:02.223119 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:02.248945 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:22:02.269315 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:22:02.333213 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:02.382587 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:02.385545 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:22:02.548060 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:02.558732 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:02.568193 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:02.589313 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:02.599300 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:02.600175 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:02.630543 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:02.734499 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:02.742687 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:02.770205 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:02.775907 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:02.797425 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:02.809479 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:02.901031 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:02.977986 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:02.984287 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:03.015396 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:03.142933 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:03.256227 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:03.338114 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:03.377498 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:03.382133 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:03.446252 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:03.476032 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:03.525918 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:03.550259 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:03.583985 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:03.781605 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:03.835819 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:22:03.874922 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:03.875003 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:03.963654 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:03.978407 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:04.059376 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.072595 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:04.081193 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.085590 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.106336 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:04.162704 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.223825 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.331661 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.370671 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:04.382184 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.429379 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:04.459019 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:04.490388 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.490682 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:04.491362 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240503 08:22:04.546789 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:04.547286 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.554684 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.576162 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.594393 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:04.663787 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.672953 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.680090 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:04.778048 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.833267 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.873610 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:04.897342 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:04.934619 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:04.963575 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:05.038280 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:05.115145 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:05.115581 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:05.148824 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:05.252192 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:05.276167 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:05.366174 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:05.434206 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:05.452533 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:05.457268 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:22:05.473062 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:05.501166 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:05.537670 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:05.554311 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:05.595989 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:05.661952 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:05.684844 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:05.727252 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 08:22:05.742803 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:05.867560 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:05.895884 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:05.940795 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:06.021834 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:06.062304 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 08:22:06.074502 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:22:06.075752 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:06.121569 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:06.256784 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:06.264756 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:06.297111 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:06.335209 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:06.411685 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:06.444599 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:06.561182 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:06.584574 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:06.616189 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:06.695014 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:06.798408 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:06.833544 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:06.849639 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:06.894721 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:06.896260 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:06.922736 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:06.929461 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:06.969362 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:06.978808 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:06.987076 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:06.993446 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:07.054361 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:07.058066 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:07.067245 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:07.121836 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:07.146806 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:07.191869 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:07.231288 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:07.237025 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:07.238159 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:07.277738 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:07.323665 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:07.350350 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:07.358323 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:07.581501 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:07.653568 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:07.685752 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:07.702930 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:07.776018 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:07.834905 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:07.863094 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:07.875006 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:07.885372 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:07.920440 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:07.963299 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:08.009683 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:08.061050 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:08.154021 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:08.175192 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:08.279255 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:08.290087 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:08.309218 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:08.358996 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:08.420738 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:08.454803 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:08.494820 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:08.540721 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:08.625218 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:08.763530 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:22:08.768046 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:08.831549 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:08.848697 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:08.850328 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:08.908949 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:08.930230 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:09.048738 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:09.114423 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:09.191303 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:09.204893 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:09.215266 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:09.246945 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:09.276355 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:09.292213 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:09.328198 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:09.331895 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:09.344517 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:09.363857 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:09.382385 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:09.430305 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:09.430505 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:09.435863 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:09.444003 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:09.544559 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:22:09.545605 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:09.547549 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:09.548127 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:09.694535 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:09.713812 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:09.724499 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:09.738345 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:09.771900 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:09.807847 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:09.874853 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:09.933236 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:10.030524 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:10.122269 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:10.153796 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:10.208741 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:10.210141 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:10.231451 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:10.293843 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:10.356105 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:10.505232 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:10.521976 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240503 08:22:10.587160 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:10.591706 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:10.631489 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:10.734037 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:10.738600 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 08:22:10.780110 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:10.784965 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:10.785784 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:10.814438 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:10.827358 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:10.828676 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:10.856106 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:11.008060 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:11.067420 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:22:11.089931 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:11.196971 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:11.243727 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:11.375569 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:11.390367 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:11.398352 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:11.426142 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:11.426146 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:11.432617 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:11.507256 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:22:11.569155 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:11.600029 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:11.617121 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:11.618348 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:11.634542 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:11.657320 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:11.662603 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:11.735390 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:11.792203 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:11.886336 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:11.950342 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:12.002817 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:12.005281 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:12.007650 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:12.053968 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:12.117482 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:12.121320 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:12.127597 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:12.157176 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:12.161631 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:12.179100 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:12.193502 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:12.232154 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:12.313308 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:12.359200 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:12.508447 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:12.532205 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:12.544978 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:12.558701 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:12.599102 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:12.638103 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:12.678948 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
I20240503 08:22:12.713168 17224 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:22:12.713330 17224 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:22:12.713804 17224 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:22:12.715085 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:12.715256 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:22:12.715353 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:22:12.715601 17224 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:22:12.838263 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:12.859337 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:12.918419 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:12.962987 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:12.976516 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:13.002887 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:13.065119 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:13.152690 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:13.156599 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:13.219077 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:13.259924 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:13.268463 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:13.298261 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:13.388700 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:13.389176 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:13.464316 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:13.479830 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:13.510706 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:13.540985 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:13.730629 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:13.768330 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:13.773330 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:13.799191 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:13.924163 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:13.928836 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:22:13.960464 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:13.984375 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:14.052534 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:14.085673 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:14.095252 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:14.106585 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:14.155273 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:14.176388 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:14.178215 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:14.349618 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:14.349692 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:14.367592 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:14.431082 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:14.431622 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:14.451578 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:14.477553 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:14.486855 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:14.515202 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:14.545914 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:22:14.668068 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:14.672143 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:14.733117 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:14.743595 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:14.756634 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:14.823459 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:14.825263 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:14.939013 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:14.976944 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:15.004029 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:15.045467 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:15.078810 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:15.176802 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:15.189579 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:15.197567 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:15.227002 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:15.284896 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:15.374779 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:15.529174 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:22:15.529945 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:15.623183 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:15.635244 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:15.637050 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:15.697963 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:15.698810 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:15.714268 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:15.720443 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:15.737262 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:15.844537 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:15.846568 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:15.846732 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 08:22:15.902391 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:15.993858 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:16.003187 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:16.069411 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240503 08:22:16.070494 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:16.095033 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:16.246748 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:16.251526 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:16.309491 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:16.314848 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:16.345566 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:16.406733 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:16.444073 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:16.496553 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:16.503196 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:16.563371 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:16.587962 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:22:16.648388 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:16.690346 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:16.768653 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:16.834695 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:16.867398 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:16.882599 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:16.882611 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:16.906476 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:16.914697 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:17.040969 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:17.088927 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:17.114122 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:17.191151 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:17.203079 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:17.254818 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:17.285012 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:17.289485 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:17.321204 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:17.375424 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:17.376109 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:17.442464 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:17.464818 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:17.535171 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:17.609334 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:17.627885 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:17.680497 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:17.752096 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:17.754586 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:17.776017 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:17.800297 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:17.818444 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:17.961007 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:18.004124 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:18.052187 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:18.124058 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:18.160339 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:18.204011 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:18.243860 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:18.360088 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:18.371690 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:18.404326 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:18.406702 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:18.452687 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:18.516268 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:18.519645 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:18.528818 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:18.646908 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:18.656958 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:18.738687 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:18.765256 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:18.787577 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:18.889739 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:18.937924 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:18.944075 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:18.991744 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:19.060297 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:19.073014 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:19.104429 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:19.112013 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:19.123185 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:19.144390 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:22:19.272594 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:19.292749 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:19.422044 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:19.441591 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:19.455544 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:19.484541 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:19.506004 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:19.580055 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:22:19.583680 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:19.645300 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:19.677822 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:19.687917 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:19.700641 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:19.726765 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:19.775979 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:19.813575 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:19.846443 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:19.853358 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:19.958403 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:20.003710 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:20.016302 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:20.037474 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:20.045280 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:20.050554 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:20.123343 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:20.125928 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:20.277319 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:20.540146 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:20.549382 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 125 similar messages]
W20240503 08:22:20.552816 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:20.571427 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:20.599925 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:20.607894 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:20.636947 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:20.693104 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:20.705132 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:20.726907 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:20.878927 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:20.888002 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 08:22:20.888775 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:20.959290 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:20.964454 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:20.983278 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:20.983829 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:21.002604 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:21.018172 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:21.143404 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:22:21.209414 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:21.221474 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:21.231036 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:21.325907 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:21.375038 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:21.410264 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:21.413324 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:22:21.416257 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:21.474655 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:21.480523 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:21.615175 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:21.647727 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:21.667258 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:21.714330 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:21.716945 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:21.733503 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:21.796316 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:21.811903 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:21.835604 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:21.865450 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:21.882916 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:22.086295 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:22.116954 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:22.144968 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:22.179801 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:22.231217 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:22.250988 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:22.284480 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:22.286409 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:22.306746 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:22.385407 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:22.422809 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:22.429728 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:22.474364 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:22.496022 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:22.510037 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:22.565892 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:22.597472 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:22.649971 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:22.696843 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:22.737931 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:22.833923 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:22.995848 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:23.018306 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:23.041548 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:23.082540 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:23.107542 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:23.254315 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:23.269865 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:23.288797 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:23.325311 28219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:22:23.326467 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d60a162155504975a39c20eff88341e1: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:22:23.404518 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:23.411737 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:23.414984 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:23.425904 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:23.448012 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:23.461576 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:23.468583 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:23.525290 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:23.617106 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:23.713260 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:23.734735 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:23.801445 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:22:23.817551 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:23.932153 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:23.941268 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:23.956022 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:23.974965 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:23.975095 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:24.012004 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:24.138762 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:24.145864 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:24.236052 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:24.254613 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:24.377883 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:24.410194 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:24.460546 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:24.465966 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:24.537259 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:24.552654 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:24.573611 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:24.573626 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:24.600553 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:24.686014 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:22:24.687058 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:24.723742 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:24.753765 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:24.769332 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:24.807736 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:24.853550 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:24.876991 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:24.892382 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d13c33b8712e4c488abb99b73ff42b86: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:22:24.969763 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:25.025043 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:25.096500 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:25.111467 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:25.121248 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:25.127835 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:25.136691 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:25.172361 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:25.189818 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:25.238197 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:25.408964 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:25.416172 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:25.506131 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:25.619904 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:22:25.654883 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:25.683475 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:25.731508 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:25.798864 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:25.825897 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:25.830252 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:25.873598 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:25.911229 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:25.923110 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 08:22:25.923928 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:25.974802 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:25.993333 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:26.032282 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:26.066411 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d44335d8be7245b1bb52b2eab9e42c18: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:22:26.078359 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:26.103464 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:26.112282 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:26.236991 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:26.240691 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:22:26.276443 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:26.303592 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:22:26.315776 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:26.417770 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:26.436064 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:26.447685 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:26.469071 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:26.605954 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:26.611011 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:26.654156 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:26.693184 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:26.855516 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:26.869738 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:26.883703 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:26.899950 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:26.972541 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:26.996197 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:27.047628 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:27.058276 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:27.058678 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:27.073011 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:27.088085 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:27.098474 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:27.204293 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:27.225553 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:27.278755 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:27.298625 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:27.351430 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:27.384075 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:27.384840 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:27.387863 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:27.409279 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:27.503144 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:27.516328 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:27.584550 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:27.641291 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:27.652737 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:27.740067 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:27.878472 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:27.881237 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:27.930816 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:27.935416 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:28.107821 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:28.125049 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:28.134675 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:28.141094 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:28.225423 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:28.321175 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:28.351073 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:28.409582 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:28.410120 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:28.490346 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:28.523149 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:28.588620 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:28.636737 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:28.673633 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:28.698431 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:28.743021 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:28.750178 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:22:28.805603 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:28.867877 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:28.914511 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:28.920646 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:28.931869 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:28.978631 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:29.085325 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:29.102982 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:29.103092 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:29.223541 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:29.224205 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:29.329823 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:29.352612 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:29.389901 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:29.400239 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:29.409986 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:29.560310 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:29.583622 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:29.666162 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:29.667918 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:29.669395 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:29.681020 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:29.689182 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:29.690816 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:29.733692 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:22:29.776278 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:29.806689 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:29.831296 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:29.867194 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:29.871903 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:29.891098 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:29.983014 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:30.140172 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:30.163070 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:30.165583 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:30.189505 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:30.204185 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:30.205492 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:30.388795 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:30.394522 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:30.481204 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:30.481331 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:30.508214 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:30.513880 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:30.574201 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:30.626453 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:22:30.709558 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:30.789127 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:30.866174 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:30.874109 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:30.930662 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 08:22:30.978071 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:30.979754 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:22:30.989218 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:31.003147 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:31.021800 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:31.076395 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:31.099097 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:31.205080 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:31.212528 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:31.245934 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:22:31.268340 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:31.345757 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:31.363734 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:31.365137 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:31.371099 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:31.435693 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:31.445605 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:31.455238 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:31.555763 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:31.752980 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:31.798363 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:22:31.821597 18293 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:22:31.821806 18293 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:22:31.822574 18293 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:22:31.823952 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:31.824115 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:22:31.824190 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:22:31.824417 18293 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:22:31.830451 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:31.850780 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:31.854936 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:31.963721 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:32.009985 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:32.041625 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:32.042124 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:32.045765 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:32.057462 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:32.114312 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:32.150730 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:32.159463 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:32.194591 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:32.220151 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:32.270077 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:32.287139 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:32.295040 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:32.321188 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:32.340943 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:32.407567 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:32.470767 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:32.550226 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:32.610263 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:32.740435 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:32.747654 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:32.750530 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:32.886134 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:32.926235 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:33.014408 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:33.057927 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:33.061021 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:33.072631 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:33.106884 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:33.110765 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:33.116556 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:22:33.125238 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:33.167273 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:33.223583 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:33.243678 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:33.355449 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:33.387816 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:33.423772 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:33.437139 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:33.550227 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:33.556129 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:33.576128 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:33.585156 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:33.615581 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:33.784548 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:33.784619 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:33.802616 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:33.820333 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:33.829140 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:33.926445 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:34.002755 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:34.214085 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:34.230427 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:34.325299 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:34.346521 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:34.347157 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:34.379868 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:34.395198 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:34.504106 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:34.514299 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:34.518240 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:34.519595 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:34.544085 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:34.573736 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:34.633055 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:34.638841 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:34.680168 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:34.690845 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:34.760732 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:34.790477 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240503 08:22:34.816936 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:34.859633 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:34.916224 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:34.962976 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:34.991468 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:34.994999 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:35.009097 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:35.032259 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:35.238319 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:35.249693 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:35.337054 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:35.405766 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:35.434722 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:35.455808 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:35.500461 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:35.501387 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:35.510917 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:35.531868 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:35.548520 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:35.610373 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:35.669564 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:22:35.670375 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:35.716938 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:35.745819 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:35.777946 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:22:35.831689 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:35.849035 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:35.868659 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:35.883488 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:35.955346 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:35.958302 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 08:22:35.959161 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:35.974092 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:36.128695 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:36.149296 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:36.203388 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:36.261579 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:22:36.320202 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:36.328802 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:36.343214 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:36.350832 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:36.396186 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:36.458413 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:36.570077 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:36.636350 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:36.659729 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:36.661623 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:36.669450 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:36.707252 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:36.822651 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:36.842547 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:36.845427 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:36.896907 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:36.967571 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:36.988685 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:22:37.041303 18702 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:22:37.041544 18702 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:22:37.042289 18702 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:22:37.042853 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:37.043640 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:37.044548 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:37.044732 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:22:37.044809 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:22:37.045069 18702 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:22:37.075412 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:37.173852 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:37.192699 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:37.255753 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:37.258798 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:37.322280 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:37.389467 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:37.411278 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:37.419862 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:37.551301 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:37.567234 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:37.661880 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:37.683655 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:37.703116 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:37.738978 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:37.843907 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:37.880215 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:37.907524 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:37.917657 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:37.950421 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:37.974220 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:38.043555 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:38.054627 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:38.057144 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:22:38.070112 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:38.085119 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:38.121304 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:38.194279 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:38.206471 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:38.256500 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:38.386399 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:38.434978 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:38.447981 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:38.523846 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:38.536666 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:38.573427 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:38.651959 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:38.698458 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:38.788522 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:38.848104 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:38.893690 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:38.912966 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:39.024566 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:39.053647 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:39.062242 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:39.079497 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:39.090480 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:39.145092 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:39.161453 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:39.164790 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:39.366641 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:39.447877 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:39.452644 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:39.479480 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:39.485989 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:39.561215 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:39.588413 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:39.637212 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:39.644152 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:39.720613 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:39.722117 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:39.731323 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:39.807492 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:39.820257 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:39.834594 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 08:22:39.835455 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:39.852725 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:40.056500 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:40.119396 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:40.158042 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:40.170743 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:40.219777 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:40.226172 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:40.228129 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:40.309477 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:40.314117 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:40.319823 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:40.336802 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:40.466418 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:40.476305 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:40.496778 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:40.521301 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:40.531750 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:40.541042 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:40.575906 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:40.622898 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:40.632232 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:40.638397 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:40.693537 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:22:40.706676 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 08:22:40.707425 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:40.765236 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:40.891474 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:40.959054 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 08:22:40.960206 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:41.041725 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:41.069715 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:41.161451 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:41.168772 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:41.249035 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:41.264623 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:41.280192 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:41.297430 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240503 08:22:41.434619 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:41.436116 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:41.451418 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:41.494935 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:41.647182 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:41.647717 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:41.664527 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:41.703994 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:41.744397 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:41.754607 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:41.849398 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:41.890581 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:41.897574 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:41.971400 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:42.023506 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:42.053900 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:42.092437 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:42.172820 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:42.224094 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:42.276847 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:42.341986 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:42.352690 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:42.355036 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:42.371196 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:42.392379 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:42.459563 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:42.519995 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:42.528110 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:42.608561 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:42.635540 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:42.700732 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:42.756515 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:42.810410 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:42.830070 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:42.832369 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:42.930184 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:42.936888 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:42.940799 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:42.941295 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:42.953423 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:42.965703 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:42.985723 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:43.047297 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:43.066972 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:43.085310 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:43.102245 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:22:43.108211 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:43.142612 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:43.240757 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:43.333817 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:43.363361 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
I20240503 08:22:43.418850 18702 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:22:43.419050 18702 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:22:43.419700 18702 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:22:43.420958 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:43.421114 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:22:43.421177 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:22:43.421375 18702 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:22:43.492776 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:43.521906 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:43.546626 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:43.675894 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:43.750483 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:43.769276 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:43.789572 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:43.854841 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:43.880810 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:43.885031 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:44.061200 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:44.064329 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:44.104722 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:44.200086 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:44.215296 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:44.269457 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:44.301566 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:44.320509 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:44.428905 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:44.429621 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:44.452603 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:44.557579 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:44.562916 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:44.581074 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:44.641220 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:44.676520 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:44.679251 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:44.781049 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:44.782289 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:44.805887 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:44.832705 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:44.844098 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:44.857358 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:44.860914 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:44.913822 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:22:45.090412 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:45.180028 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:45.182631 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:45.187629 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:45.191804 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:45.212052 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:45.242635 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:45.248414 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:45.252346 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:22:45.254355 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:45.259368 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:45.274721 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:45.287256 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:45.303970 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:45.356633 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:22:45.401645 18702 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:22:45.401829 18702 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:22:45.402478 18702 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:22:45.403980 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:45.404119 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:22:45.404192 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:22:45.404403 18702 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:22:45.440219 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:45.600261 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:45.609968 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:45.617316 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:45.638100 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:45.693411 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:45.725708 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:22:45.753472 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:45.782171 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:45.907682 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:46.011487 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:46.016636 28568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 08:22:46.143852 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:46.169224 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:46.198308 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:46.229823 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:46.245622 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:46.246793 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:46.247583 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:46.343758 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:22:46.344723 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:46.518916 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:46.580721 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:46.609582 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:46.642203 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:46.650013 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:46.653967 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:46.692682 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:46.716504 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:46.782009 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:46.875007 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:47.042269 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:47.060528 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:47.071311 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:47.095566 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:47.099939 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:47.112226 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:47.115811 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:47.141474 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:47.158774 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:47.263226 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:47.265218 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:47.370959 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:47.390136 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:47.394857 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:47.503341 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:47.556391 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:47.578825 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:22:47.603717 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:47.661861 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:47.669204 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:47.742733 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:47.777211 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:47.794370 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:47.868731 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:47.893615 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:47.933346 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:47.984597 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:47.987776 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:48.031592 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:48.037887 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:48.044270 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:48.056793 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:48.111692 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:48.148748 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:48.175856 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:48.252015 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:48.335711 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:48.349483 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:48.485028 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:48.575542 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:48.596838 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:48.642676 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:48.684826 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:48.703188 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:48.720470 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:48.759229 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:48.945327 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:48.974155 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:49.003580 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:49.092716 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:49.098423 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:49.180269 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:49.243192 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:49.258971 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:49.272135 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:49.349951 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:49.420809 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:49.460108 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:49.471920 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:49.500406 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:49.512629 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:49.512703 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:49.620278 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:49.629624 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:49.667568 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:49.713783 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:49.727449 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:49.781080 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:49.848958 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:49.852952 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:49.952030 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:49.974802 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:50.000186 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 08:22:50.000840 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:50.022431 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:50.049855 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:50.121178 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:50.218079 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:50.255303 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:50.287979 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:50.343300 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:50.365080 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:50.385171 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:50.411579 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:50.455140 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:22:50.496816 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:50.511538 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:50.533365 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:50.559856 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:50.595214 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:50.623122 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:50.629449 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:50.667187 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:50.727483 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:22:50.748975 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:50.800066 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:50.805810 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:50.844163 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:51.000448 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:51.035679 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:51.081511 28568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 08:22:51.104214 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:51.120574 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:51.152369 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:51.155118 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:51.198752 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:51.213585 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:51.234232 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:51.403949 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 08:22:51.467746 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:51.494328 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:51.608294 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:51.615096 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:51.684201 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:51.726075 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:51.734248 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:51.755623 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:51.797762 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:51.866024 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:51.868042 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:52.033608 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:52.098842 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:52.104044 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:52.176400 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:52.177996 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:52.183598 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:52.221769 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:52.322006 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:52.361726 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:52.363153 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:52.424415 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:52.425701 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:52.452391 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:52.576913 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:52.599431 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:52.622689 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:52.633744 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:52.679772 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:52.685880 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:52.791342 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:52.839217 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:52.851219 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:52.870927 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:52.878363 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:52.901525 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:52.910085 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:52.919421 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:52.920855 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:22:52.987907 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:53.006040 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:53.015350 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:53.109354 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:53.144248 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:53.227412 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:53.261507 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:53.269606 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:53.279472 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:53.306329 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:53.444608 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:53.474608 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:53.509477 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:53.580538 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:53.645550 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:53.664903 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:53.673760 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:53.729979 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:53.781183 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:53.847316 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:53.859498 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:53.933768 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:54.173705 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:54.195227 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:54.247359 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:54.251577 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:54.353823 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:54.365041 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:54.430755 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:54.456859 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:54.493315 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:54.671591 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:54.714661 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:54.726477 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:54.733492 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:54.766171 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:54.768906 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:54.801343 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:54.816980 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:54.833184 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:54.890378 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:54.890720 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:54.893687 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:54.940495 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:54.954944 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:55.028717 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:22:55.095263 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:55.126897 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:55.138659 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:55.159458 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:55.163483 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:55.174641 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:55.245882 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:55.293658 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:22:55.401154 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:55.402395 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:55.410988 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:55.470574 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:55.522773 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:55.536363 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:55.558959 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:55.596235 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:55.606352 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:55.642364 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:55.692971 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:55.734620 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240503 08:22:55.735625 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:55.786104 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:55.842744 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:55.872956 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:55.971890 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:55.979362 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:56.030546 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:56.043797 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:56.131882 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:56.156472 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:56.170840 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:56.199421 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:56.268520 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:56.278860 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 08:22:56.370294 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:56.406291 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:56.410728 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 08:22:56.521486 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:56.706250 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:56.706657 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:56.721282 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:56.728737 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:56.738009 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:56.816363 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:56.920022 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:56.996944 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:57.004024 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:57.135795 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:57.180503 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:57.186551 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:57.291522 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:57.307912 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:57.312861 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:57.338994 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:57.371990 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:57.411129 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:57.412880 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:57.424021 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:57.480187 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:57.540112 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:57.540789 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:57.603454 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:57.608505 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:57.633752 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:57.684545 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:57.737828 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:22:57.737799 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:57.795233 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:57.869829 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:57.882390 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:22:57.900612 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:57.911164 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:57.968585 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:58.004998 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:22:58.034529 20347 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:22:58.034713 20347 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:22:58.035351 20347 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:22:58.037055 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:58.037209 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:22:58.037293 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:22:58.037518 20347 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:22:58.081266 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:58.114730 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:58.184845 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:22:58.196951 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:58.246016 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:58.295060 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:58.347371 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:22:58.374377 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:58.396811 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:58.403767 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:58.411777 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:58.462864 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:58.521876 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:58.608294 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:58.625413 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:58.647471 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:58.680660 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:58.702104 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:58.812147 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:58.848795 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:58.935227 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:59.089561 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:59.143831 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:59.162559 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:59.297056 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:59.312599 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:59.314548 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:59.385746 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:59.400146 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:59.541553 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:59.552832 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:59.574404 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:22:59.631196 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:59.642760 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:59.783197 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:22:59.832188 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:22:59.840552 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:59.847000 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:22:59.884438 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:59.885380 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:22:59.926509 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:22:59.939841 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:22:59.972285 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:00.035272 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:23:00.053450 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:23:00.069877 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:00.074184 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:00.076656 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:23:00.087864 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:23:00.177634 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:00.225031 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:23:00.288897 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:23:00.296712 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:00.297291 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:00.315816 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:00.325459 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:00.374382 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:00.448813 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:00.491057 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:00.562202 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:23:00.581990 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:00.583738 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:00.644482 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:00.713279 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:23:00.717720 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:00.774433 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:23:00.775432 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:00.799564 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:23:00.856959 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:00.879185 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:00.921430 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:00.941845 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:23:01.023483 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:23:01.030181 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:01.094502 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:01.137567 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:23:01.200218 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:01.233913 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:23:01.267105 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:01.296118 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:01.322206 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 08:23:01.416793 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:23:01.434906 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:01.496214 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:01.570422 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:01.666354 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:01.669654 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:01.715334 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:01.857210 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:01.872066 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:01.882741 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:01.891700 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:01.980967 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:23:01.982448 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:02.003798 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:02.049948 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:02.170306 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:02.287477 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:02.315888 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:02.373966 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:02.442915 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:02.453188 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:02.457259 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:02.458737 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:02.475389 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:02.488768 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:23:02.510819 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:02.575728 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:02.637703 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:02.671941 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:02.700485 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:02.743333 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:02.746160 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:23:02.783800 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:02.794479 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:02.812144 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:02.913128 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:02.967037 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:03.009236 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:03.044521 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:03.091357 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:03.107370 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:03.114037 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:03.124387 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:03.225672 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:03.275525 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:03.317991 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:03.346612 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:03.426956 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:03.455264 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:03.460467 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:03.489480 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:03.507881 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:03.530777 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:03.603195 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:03.691303 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:03.783583 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:03.914069 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:03.925616 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:03.942206 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:03.990285 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:04.093726 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:04.109720 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:04.128995 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:04.146318 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:04.212656 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:04.257887 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:04.277279 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:04.331555 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:04.366451 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:04.578038 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:04.579779 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:04.584190 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:04.661492 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:04.662300 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:04.724721 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:04.758119 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:04.784291 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:04.868204 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:04.881086 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:04.895148 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:04.952822 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:04.983454 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:05.055122 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:23:05.066859 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:05.078187 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:05.102260 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:05.126704 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:23:05.145750 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:05.210024 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:05.222302 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:05.242884 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:05.260643 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:05.267781 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:05.294457 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:05.339332 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:05.378047 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:05.444296 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:05.472278 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:05.474896 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:05.537981 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:05.617810 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:05.655827 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:05.702487 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:05.719655 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:05.761725 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:05.773172 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:05.828130 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:23:05.882129 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:05.921351 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:05.946436 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:06.034950 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:06.133234 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:06.153810 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:06.223570 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:06.239326 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:06.300482 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:06.318670 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:06.356809 28568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 08:23:06.359768 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:06.423977 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:23:06.578980 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:06.641566 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:06.682482 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:06.736671 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:06.778046 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:06.778963 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:06.784245 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:06.784613 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:06.878041 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:06.890059 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:06.932366 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:07.129940 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:07.166554 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:07.174984 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:07.213541 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:07.236308 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:07.254273 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:07.265919 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:07.268074 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:07.305853 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:07.414213 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:07.506532 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:07.528419 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:07.549628 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:23:07.556741 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:07.661119 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:07.663760 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:07.670871 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:07.685575 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:07.687981 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:07.690467 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:07.699329 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:07.819991 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:07.823016 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:07.839895 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:07.887095 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:07.897768 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:07.962157 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:07.966701 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:08.131971 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:08.200275 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:08.292179 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:08.295046 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:08.341535 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:08.445497 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:08.461484 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:08.490579 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:08.491953 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:08.503147 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:08.568890 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:08.568939 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:08.591482 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:08.723018 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:08.885890 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:08.909586 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:09.033272 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:09.074026 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:09.101330 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:09.122000 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:09.237275 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:09.264127 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:09.305514 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:09.359549 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:09.372032 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:09.408448 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:09.452419 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:09.481343 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:09.493909 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:09.521113 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:09.673789 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:09.718124 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:09.789170 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:09.816506 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:09.834555 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:09.883148 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:09.958889 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:09.963435 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:09.966140 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:09.989004 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:09.990345 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:23:10.017995 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:10.072862 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:10.078914 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240503 08:23:10.115432 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:10.197974 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:10.227160 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:10.230458 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:10.271340 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:10.312115 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:10.341840 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:10.345373 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:10.365938 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:10.372606 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:10.399441 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:10.453611 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:10.473378 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:10.525059 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:10.619513 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:10.634251 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:10.701967 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:10.706877 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:10.728567 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:10.735589 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:10.870918 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:10.907392 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:10.969501 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:23:10.970362 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:10.988418 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:10.988407 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:10.988704 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:11.031312 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:11.063066 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:11.157497 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:11.168108 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:11.350107 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:11.364863 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 08:23:11.453786 28698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:23:11.575919 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:11.579836 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:11.598544 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:11.716518 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:11.767903 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:11.795708 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:11.877321 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:11.902676 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:11.931910 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:11.950773 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:11.994232 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:12.106609 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:12.144953 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:12.153702 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:12.158077 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:12.182878 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:12.184181 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:12.186141 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:12.236500 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:12.302515 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:12.519255 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:12.537225 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:12.553383 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:12.562238 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:12.562238 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:12.666220 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:12.672926 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:23:12.675020 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:12.681840 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:12.689725 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:12.721287 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:12.792642 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:12.810420 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:12.813764 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:12.873343 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:12.874557 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:12.911001 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:12.927551 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:12.961086 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:13.052002 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:13.075662 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:13.131385 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:13.177799 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:13.212246 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:13.241309 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:13.254629 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:13.284955 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:13.289122 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:13.361976 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:13.371297 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:13.378782 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:13.437932 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:13.460289 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:13.594673 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:13.600764 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:13.688318 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:13.814746 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:13.910452 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:13.988411 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:14.005762 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:14.064396 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:14.068948 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:14.150593 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:14.243276 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:14.362066 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:14.400961 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:14.427165 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:14.448498 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:14.479255 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:14.520325 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:23:14.620922 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:14.624681 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:14.669034 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:14.693985 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:14.730257 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:14.771183 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:14.780208 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:14.811056 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:14.946931 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:14.965020 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:15.026110 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:15.040547 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:15.042675 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:15.055227 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:15.068630 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:23:15.167656 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:15.198865 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:23:15.219597 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:15.270661 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:15.343873 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:15.344481 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:15.349164 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:15.353483 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:15.354707 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:15.376675 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:15.389776 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:15.394177 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:15.422291 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:15.438601 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:15.531033 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:15.540215 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:15.674957 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:15.748373 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:23:15.765336 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:15.771464 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:23:15.861107 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:15.911360 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:15.923907 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:15.925700 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:15.936340 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:23:15.944026 20903 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:23:15.944175 20903 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:23:15.944725 20903 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:23:15.945885 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:15.946112 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:23:15.946188 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:23:15.946429 20903 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:23:15.949362 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:16.000695 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:16.014508 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:16.025560 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:16.034360 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:23:16.201148 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:16.311580 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:16.340004 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:16.394279 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:16.418308 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:16.506593 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:23:16.507993 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:16.524791 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 08:23:16.587538 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:16.599949 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:16.611847 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:16.651604 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:16.717378 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:16.773806 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:16.849371 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 08:23:16.864145 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:16.906352 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:16.949080 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:16.999294 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:17.102088 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:17.289732 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:17.291524 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:17.330885 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:23:17.337616 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:17.359937 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:17.370513 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:17.404145 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:17.418814 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:23:17.460506 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:17.488185 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:17.491565 28698 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:23:17.575381 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:23:17.587242 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:17.610885 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:17.630415 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:23:17.634047 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:17.717525 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:17.728782 21503 consensus_queue.cc:579] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Leader has been unable to successfully communicate with peer b29361eb4afa4ab3af7d8019d8a5042d for more than 300 seconds (300.059s)
W20240503 08:23:17.731577 28308 consensus_peers.cc:462] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
I20240503 08:23:17.737881 28373 consensus_queue.cc:237] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:23:17.740324 28502 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 08:23:17.740705 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:17.741060 21751 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:23:17.743564 21750 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:23:17.744172 28502 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:23:17.747695 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 173376257c4946b5a801a27e4dffe90d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:17.749532 28228 catalog_manager.cc:5484] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 9 to 10, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 08:23:17.754834 28218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:23:17.755725 28373 consensus_queue.cc:237] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } }
W20240503 08:23:17.758805 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095): Couldn't send request to peer 93f7e733cec9415eb422b0a7ba8621cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 173376257c4946b5a801a27e4dffe90d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:17.758800 28502 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:23:17.759541 21503 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:23:17.759742 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:23:17.761246 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
I20240503 08:23:17.761979 21503 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } } }
I20240503 08:23:17.762571 28502 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } } }
I20240503 08:23:17.768127 28228 catalog_manager.cc:5484] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 10 to 11, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:17.768102 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 173376257c4946b5a801a27e4dffe90d with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 08:23:17.769430 28218 catalog_manager.cc:4573] TS b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 173376257c4946b5a801a27e4dffe90d: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
I20240503 08:23:17.898424 20903 consensus_queue.cc:579] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Leader has been unable to successfully communicate with peer c8375219addc48fa99ac474ed1c9aef9 for more than 300 seconds (300.023s)
W20240503 08:23:17.898689 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:17.907419 28762 consensus_queue.cc:237] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:17.910233 28631 raft_consensus.cc:1261] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:23:17.910326 28699 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): Couldn't send request to peer 4e4a1242f3714d03a6c9ee51dd63929c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e9537f30aaf45adbc2cc4a784009de7. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:23:17.910585 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:17.910964 21776 consensus_queue.cc:1035] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:23:17.913198 20903 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:17.913775 28631 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:17.917205 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6e9537f30aaf45adbc2cc4a784009de7 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:17.917793 28232 catalog_manager.cc:5484] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 6 to 7, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:17.938393 21749 consensus_queue.cc:579] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Leader has been unable to successfully communicate with peer b29361eb4afa4ab3af7d8019d8a5042d for more than 300 seconds (300.181s)
W20240503 08:23:17.940124 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:17.948853 21566 consensus_queue.cc:579] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Leader has been unable to successfully communicate with peer c8375219addc48fa99ac474ed1c9aef9 for more than 300 seconds (300.074s)
W20240503 08:23:17.950500 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:17.966404 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:17.977062 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:23:18.001761 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:23:18.003998 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 08:23:18.006964 21503 consensus_queue.cc:579] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Leader has been unable to successfully communicate with peer bea2145b8e24436392b18ec3cfe90b93 for more than 300 seconds (300.135s)
I20240503 08:23:18.033856 20903 consensus_queue.cc:579] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Leader has been unable to successfully communicate with peer c8375219addc48fa99ac474ed1c9aef9 for more than 300 seconds (300.065s)
I20240503 08:23:18.038017 28762 consensus_queue.cc:237] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } }
I20240503 08:23:18.040426 28631 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:23:18.040982 20903 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 08:23:18.041289 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.043479 21776 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:18.044104 28631 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:18.046607 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6e1b9f104ea471cab7527bd56433327 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 08:23:18.048709 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:23:18.049496 28228 catalog_manager.cc:5484] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 13 to 14, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:18.055066 28762 consensus_queue.cc:237] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
W20240503 08:23:18.057371 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): Couldn't send request to peer 4e4a1242f3714d03a6c9ee51dd63929c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6e1b9f104ea471cab7527bd56433327. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.057801 28631 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
W20240503 08:23:18.058022 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.058362 21776 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:23:18.060617 21754 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:18.061205 28631 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:18.063717 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c6e1b9f104ea471cab7527bd56433327 with cas_config_opid_index 14: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:18.065788 28228 catalog_manager.cc:5484] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 14 to 15, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:23:18.075368 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:18.137676 21782 consensus_queue.cc:579] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Leader has been unable to successfully communicate with peer b29361eb4afa4ab3af7d8019d8a5042d for more than 300 seconds (300.295s)
I20240503 08:23:18.145937 21503 consensus_queue.cc:579] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Leader has been unable to successfully communicate with peer c8375219addc48fa99ac474ed1c9aef9 for more than 300 seconds (300.233s)
I20240503 08:23:18.164227 21780 consensus_queue.cc:579] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Leader has been unable to successfully communicate with peer b29361eb4afa4ab3af7d8019d8a5042d for more than 300 seconds (300.172s)
W20240503 08:23:18.170797 28698 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:18.279304 21781 consensus_queue.cc:579] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Leader has been unable to successfully communicate with peer b29361eb4afa4ab3af7d8019d8a5042d for more than 300 seconds (300.440s)
W20240503 08:23:18.281051 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:18.287746 28631 consensus_queue.cc:237] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } }
I20240503 08:23:18.289397 21503 consensus_queue.cc:579] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Leader has been unable to successfully communicate with peer c8375219addc48fa99ac474ed1c9aef9 for more than 300 seconds (300.415s)
W20240503 08:23:18.291442 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095): Couldn't send request to peer 93f7e733cec9415eb422b0a7ba8621cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d60a162155504975a39c20eff88341e1. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:23:18.292069 28307 consensus_peers.cc:462] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:18.292156 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:23:18.292349 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.292620 28373 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:23:18.293159 21818 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:23:18.293514 28374 consensus_queue.cc:237] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } }
I20240503 08:23:18.295646 28762 raft_consensus.cc:1261] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:23:18.296169 21751 consensus_queue.cc:1035] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:23:18.296691 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.296510 21781 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } } }
I20240503 08:23:18.297092 28373 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } } }
I20240503 08:23:18.297490 21780 consensus_queue.cc:579] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Leader has been unable to successfully communicate with peer c8375219addc48fa99ac474ed1c9aef9 for more than 300 seconds (300.140s)
I20240503 08:23:18.303346 21750 raft_consensus.cc:2932] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:23:18.303977 28762 raft_consensus.cc:2932] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:23:18.306061 28217 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d60a162155504975a39c20eff88341e1 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:18.306162 28228 catalog_manager.cc:5484] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 9 to 10, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:18.306660 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ede24a27608a4cf196dd46f85c4163e1 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:18.311940 28232 catalog_manager.cc:5484] T ede24a27608a4cf196dd46f85c4163e1 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 7 to 8, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "bea2145b8e24436392b18ec3cfe90b93" member_type: VOTER last_known_addr { host: "127.20.107.135" port: 45937 } attrs { promote: false } } }
I20240503 08:23:18.330811 21823 ts_tablet_manager.cc:902] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:23:18.332175 21823 tablet_copy_client.cc:308] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:23:18.332964 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet 173376257c4946b5a801a27e4dffe90d from peer 93f7e733cec9415eb422b0a7ba8621cb ({username='slave'} at 127.20.107.132:50515)
I20240503 08:23:18.333109 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet 173376257c4946b5a801a27e4dffe90d from peer 93f7e733cec9415eb422b0a7ba8621cb at {username='slave'} at 127.20.107.132:50515: session id = 93f7e733cec9415eb422b0a7ba8621cb-173376257c4946b5a801a27e4dffe90d
I20240503 08:23:18.335444 28394 tablet_copy_source_session.cc:205] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:18.337239 21823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 173376257c4946b5a801a27e4dffe90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:18.341885 21776 consensus_queue.cc:579] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Leader has been unable to successfully communicate with peer c8375219addc48fa99ac474ed1c9aef9 for more than 300 seconds (300.152s)
I20240503 08:23:18.342154 21823 tablet_copy_client.cc:783] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:18.345934 28762 consensus_queue.cc:237] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
W20240503 08:23:18.353022 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.353116 28373 raft_consensus.cc:1261] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:23:18.353647 21776 consensus_queue.cc:1035] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:23:18.354615 28699 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): Couldn't send request to peer 716ffbb2a00b4f43b03e5cedcc9d302a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 564fb1b51abd4f8da0372ec99e3c599c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.355293 21823 tablet_copy_client.cc:647] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:18.356933 21754 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:23:18.357545 21823 tablet_copy_client.cc:523] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:18.360064 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 564fb1b51abd4f8da0372ec99e3c599c with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:18.360237 28373 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:23:18.360983 21823 tablet_bootstrap.cc:492] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:23:18.366003 28232 catalog_manager.cc:5484] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 6 to 7, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:18.370795 21780 consensus_queue.cc:579] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Leader has been unable to successfully communicate with peer c8375219addc48fa99ac474ed1c9aef9 for more than 300 seconds (300.187s)
I20240503 08:23:18.373313 21801 consensus_queue.cc:579] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Leader has been unable to successfully communicate with peer b29361eb4afa4ab3af7d8019d8a5042d for more than 300 seconds (300.335s)
W20240503 08:23:18.377583 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:18.382123 28502 consensus_queue.cc:237] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:18.384280 21823 tablet_bootstrap.cc:492] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:18.384727 21823 tablet_bootstrap.cc:492] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap complete.
I20240503 08:23:18.385128 21823 ts_tablet_manager.cc:1367] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.024s user 0.021s sys 0.000s
I20240503 08:23:18.385427 28631 raft_consensus.cc:1261] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:23:18.385964 21582 consensus_queue.cc:1035] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:23:18.386332 21823 raft_consensus.cc:348] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } }
I20240503 08:23:18.386678 21823 raft_consensus.cc:729] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: LEARNER
I20240503 08:23:18.386965 21823 consensus_queue.cc:260] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } }
I20240503 08:23:18.387804 21823 ts_tablet_manager.cc:1398] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:23:18.388516 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session 93f7e733cec9415eb422b0a7ba8621cb-173376257c4946b5a801a27e4dffe90d received from {username='slave'} at 127.20.107.132:50515
I20240503 08:23:18.388638 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session 93f7e733cec9415eb422b0a7ba8621cb-173376257c4946b5a801a27e4dffe90d on tablet 173376257c4946b5a801a27e4dffe90d with peer 93f7e733cec9415eb422b0a7ba8621cb
I20240503 08:23:18.389771 21582 consensus_queue.cc:579] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Leader has been unable to successfully communicate with peer c8375219addc48fa99ac474ed1c9aef9 for more than 300 seconds (300.235s)
W20240503 08:23:18.393448 28439 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): Couldn't send request to peer 4e4a1242f3714d03a6c9ee51dd63929c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:23:18.394845 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.396054 28502 consensus_queue.cc:237] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
W20240503 08:23:18.398849 28439 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): Couldn't send request to peer 4e4a1242f3714d03a6c9ee51dd63929c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8b4e398d34443b4a440f9ba48555d84. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.399932 28762 raft_consensus.cc:1261] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:23:18.400940 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.400692 21780 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:18.404572 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 745d736d43244d4f9063f61de77cb473 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:18.404606 21749 consensus_queue.cc:1035] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:23:18.405992 28635 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:18.412309 28226 catalog_manager.cc:5484] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 6 to 7, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:18.416647 21749 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
W20240503 08:23:18.418373 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:23:18.420213 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a8b4e398d34443b4a440f9ba48555d84 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:18.420579 28762 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:18.422610 28232 catalog_manager.cc:5484] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 6 to 7, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:18.422981 21780 consensus_queue.cc:579] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Leader has been unable to successfully communicate with peer b29361eb4afa4ab3af7d8019d8a5042d for more than 300 seconds (300.474s)
W20240503 08:23:18.427908 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 08:23:18.429373 21838 ts_tablet_manager.cc:902] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:23:18.431181 21838 tablet_copy_client.cc:308] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:23:18.431114 28502 consensus_queue.cc:237] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:18.431993 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 6e9537f30aaf45adbc2cc4a784009de7 from peer 4e4a1242f3714d03a6c9ee51dd63929c ({username='slave'} at 127.20.107.129:57301)
I20240503 08:23:18.432166 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 6e9537f30aaf45adbc2cc4a784009de7 from peer 4e4a1242f3714d03a6c9ee51dd63929c at {username='slave'} at 127.20.107.129:57301: session id = 4e4a1242f3714d03a6c9ee51dd63929c-6e9537f30aaf45adbc2cc4a784009de7
W20240503 08:23:18.433434 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): Couldn't send request to peer 4e4a1242f3714d03a6c9ee51dd63929c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 320d1e880958427a88e93321171cb36e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.433430 28762 raft_consensus.cc:1261] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:23:18.433935 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.433977 21780 consensus_queue.cc:1035] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:23:18.435245 28785 tablet_copy_source_session.cc:205] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:18.436053 21750 consensus_queue.cc:579] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Leader has been unable to successfully communicate with peer c8375219addc48fa99ac474ed1c9aef9 for more than 300 seconds (300.170s)
I20240503 08:23:18.436623 21749 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:18.437162 21838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9537f30aaf45adbc2cc4a784009de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:18.439915 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 320d1e880958427a88e93321171cb36e with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:18.440647 28762 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:18.442620 21838 tablet_copy_client.cc:783] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:18.443800 28226 catalog_manager.cc:5484] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 6 to 7, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:18.445669 28373 consensus_queue.cc:237] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
W20240503 08:23:18.449262 28308 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): Couldn't send request to peer 716ffbb2a00b4f43b03e5cedcc9d302a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.450423 28762 raft_consensus.cc:1261] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:23:18.450657 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.450929 21751 consensus_queue.cc:1035] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:23:18.453444 21750 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:23:18.456300 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a8a9adc3712d436c95505a3216e7607c with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:18.457098 21838 tablet_copy_client.cc:647] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:18.456676 28762 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:23:18.458212 28232 catalog_manager.cc:5484] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 6 to 7, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:18.459075 21838 tablet_copy_client.cc:523] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 08:23:18.463570 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:18.463721 21838 tablet_bootstrap.cc:492] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:23:18.482290 21838 tablet_bootstrap.cc:492] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:18.482812 21838 tablet_bootstrap.cc:492] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap complete.
I20240503 08:23:18.483135 21838 ts_tablet_manager.cc:1367] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.001s
I20240503 08:23:18.484007 21838 raft_consensus.cc:348] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:18.484207 21838 raft_consensus.cc:729] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: LEARNER
I20240503 08:23:18.484419 21838 consensus_queue.cc:260] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:18.484930 21838 ts_tablet_manager.cc:1398] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:23:18.485703 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-6e9537f30aaf45adbc2cc4a784009de7 received from {username='slave'} at 127.20.107.129:57301
I20240503 08:23:18.485826 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-6e9537f30aaf45adbc2cc4a784009de7 on tablet 6e9537f30aaf45adbc2cc4a784009de7 with peer 4e4a1242f3714d03a6c9ee51dd63929c
I20240503 08:23:18.525285 21836 consensus_queue.cc:579] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Leader has been unable to successfully communicate with peer b29361eb4afa4ab3af7d8019d8a5042d for more than 300 seconds (300.510s)
W20240503 08:23:18.527127 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:23:18.529774 28502 consensus_queue.cc:237] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:18.532761 28631 raft_consensus.cc:1261] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:23:18.533155 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): Couldn't send request to peer 4e4a1242f3714d03a6c9ee51dd63929c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da529c8f8de241d38678d1596b9402ac. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:23:18.533679 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.533749 21836 consensus_queue.cc:1035] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:23:18.536345 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(6e9537f30aaf45adbc2cc4a784009de7): 4923 bytes on disk
I20240503 08:23:18.536154 21582 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:18.536906 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(6e9537f30aaf45adbc2cc4a784009de7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240503 08:23:18.536758 28631 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:18.538929 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(173376257c4946b5a801a27e4dffe90d): 4924 bytes on disk
I20240503 08:23:18.539505 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(173376257c4946b5a801a27e4dffe90d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240503 08:23:18.542670 28232 catalog_manager.cc:5484] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 6 to 7, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:18.543784 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet da529c8f8de241d38678d1596b9402ac with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:18.614851 21582 consensus_queue.cc:579] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Leader has been unable to successfully communicate with peer c8375219addc48fa99ac474ed1c9aef9 for more than 300 seconds (300.240s)
I20240503 08:23:18.634079 21754 consensus_queue.cc:579] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Leader has been unable to successfully communicate with peer c8375219addc48fa99ac474ed1c9aef9 for more than 300 seconds (300.429s)
I20240503 08:23:18.635879 21838 ts_tablet_manager.cc:902] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:23:18.637095 21838 tablet_copy_client.cc:308] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:23:18.637770 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet c6e1b9f104ea471cab7527bd56433327 from peer 4e4a1242f3714d03a6c9ee51dd63929c ({username='slave'} at 127.20.107.129:57301)
I20240503 08:23:18.637898 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet c6e1b9f104ea471cab7527bd56433327 from peer 4e4a1242f3714d03a6c9ee51dd63929c at {username='slave'} at 127.20.107.129:57301: session id = 4e4a1242f3714d03a6c9ee51dd63929c-c6e1b9f104ea471cab7527bd56433327
I20240503 08:23:18.638536 28762 consensus_queue.cc:237] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } }
I20240503 08:23:18.640509 28631 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:23:18.640821 28785 tablet_copy_source_session.cc:205] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:18.641058 21801 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 08:23:18.641253 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.642165 21838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e1b9f104ea471cab7527bd56433327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:18.643422 21754 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:18.644034 28631 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:18.647383 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0caf86850b3841ea811b0a6ab19ae330 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:18.647159 21838 tablet_copy_client.cc:783] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:18.649709 28226 catalog_manager.cc:5484] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 13 to 14, NON_VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:18.655686 28762 consensus_queue.cc:237] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
W20240503 08:23:18.658001 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): Couldn't send request to peer 4e4a1242f3714d03a6c9ee51dd63929c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0caf86850b3841ea811b0a6ab19ae330. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:23:18.658552 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:18.659045 21838 tablet_copy_client.cc:647] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:18.660815 21838 tablet_copy_client.cc:523] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:18.664324 28631 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240503 08:23:18.664944 21801 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:23:18.665400 21838 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:23:18.666927 21754 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:18.667621 28631 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:18.670996 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0caf86850b3841ea811b0a6ab19ae330 with cas_config_opid_index 14: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 08:23:18.671553 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:23:18.673620 28232 catalog_manager.cc:5484] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 14 to 15, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:18.701951 21838 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:18.702291 21838 tablet_bootstrap.cc:492] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap complete.
I20240503 08:23:18.702512 21838 ts_tablet_manager.cc:1367] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.037s user 0.019s sys 0.004s
I20240503 08:23:18.703049 21838 raft_consensus.cc:348] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:18.703227 21838 raft_consensus.cc:729] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: LEARNER
I20240503 08:23:18.703397 21838 consensus_queue.cc:260] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:18.703881 21838 ts_tablet_manager.cc:1398] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:23:18.704941 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-c6e1b9f104ea471cab7527bd56433327 received from {username='slave'} at 127.20.107.129:57301
I20240503 08:23:18.705054 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-c6e1b9f104ea471cab7527bd56433327 on tablet c6e1b9f104ea471cab7527bd56433327 with peer 4e4a1242f3714d03a6c9ee51dd63929c
W20240503 08:23:18.718732 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:23:18.777580 21823 ts_tablet_manager.cc:902] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:23:18.778824 21823 tablet_copy_client.cc:308] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:23:18.779428 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet d60a162155504975a39c20eff88341e1 from peer 93f7e733cec9415eb422b0a7ba8621cb ({username='slave'} at 127.20.107.132:55517)
I20240503 08:23:18.779595 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet d60a162155504975a39c20eff88341e1 from peer 93f7e733cec9415eb422b0a7ba8621cb at {username='slave'} at 127.20.107.132:55517: session id = 93f7e733cec9415eb422b0a7ba8621cb-d60a162155504975a39c20eff88341e1
I20240503 08:23:18.782037 28655 tablet_copy_source_session.cc:205] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:18.782461 21892 ts_tablet_manager.cc:902] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:23:18.783509 21823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60a162155504975a39c20eff88341e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:18.785131 21892 tablet_copy_client.cc:308] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:23:18.785727 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 564fb1b51abd4f8da0372ec99e3c599c from peer 716ffbb2a00b4f43b03e5cedcc9d302a ({username='slave'} at 127.20.107.130:35911)
I20240503 08:23:18.785861 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 564fb1b51abd4f8da0372ec99e3c599c from peer 716ffbb2a00b4f43b03e5cedcc9d302a at {username='slave'} at 127.20.107.130:35911: session id = 716ffbb2a00b4f43b03e5cedcc9d302a-564fb1b51abd4f8da0372ec99e3c599c
I20240503 08:23:18.788121 28785 tablet_copy_source_session.cc:205] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:18.788319 21823 tablet_copy_client.cc:783] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:18.789047 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(c6e1b9f104ea471cab7527bd56433327): 4921 bytes on disk
I20240503 08:23:18.789494 21892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564fb1b51abd4f8da0372ec99e3c599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:18.789667 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(c6e1b9f104ea471cab7527bd56433327) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240503 08:23:18.794562 21892 tablet_copy_client.cc:783] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:18.795734 21823 tablet_copy_client.cc:647] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:18.797482 21823 tablet_copy_client.cc:523] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:18.800733 21823 tablet_bootstrap.cc:492] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:23:18.800773 21892 tablet_copy_client.cc:647] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:18.804172 21892 tablet_copy_client.cc:523] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:18.806890 28762 raft_consensus.cc:1203] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 08:23:18.807374 21892 tablet_bootstrap.cc:492] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:23:18.811990 21838 ts_tablet_manager.cc:902] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:23:18.823488 21892 tablet_bootstrap.cc:492] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:18.823966 21892 tablet_bootstrap.cc:492] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap complete.
I20240503 08:23:18.824084 21823 tablet_bootstrap.cc:492] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:18.824340 21892 ts_tablet_manager.cc:1367] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.017s user 0.017s sys 0.000s
I20240503 08:23:18.824699 21823 tablet_bootstrap.cc:492] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap complete.
I20240503 08:23:18.825037 21823 ts_tablet_manager.cc:1367] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.024s user 0.027s sys 0.000s
I20240503 08:23:18.825235 21892 raft_consensus.cc:348] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:23:18.825462 21892 raft_consensus.cc:729] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: LEARNER
I20240503 08:23:18.825667 21892 consensus_queue.cc:260] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:23:18.826208 21823 raft_consensus.cc:348] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } }
I20240503 08:23:18.826604 21823 raft_consensus.cc:729] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: LEARNER
I20240503 08:23:18.826797 21892 ts_tablet_manager.cc:1398] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:23:18.826843 21823 consensus_queue.cc:260] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } }
I20240503 08:23:18.827510 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session 716ffbb2a00b4f43b03e5cedcc9d302a-564fb1b51abd4f8da0372ec99e3c599c received from {username='slave'} at 127.20.107.130:35911
I20240503 08:23:18.827638 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session 716ffbb2a00b4f43b03e5cedcc9d302a-564fb1b51abd4f8da0372ec99e3c599c on tablet 564fb1b51abd4f8da0372ec99e3c599c with peer 716ffbb2a00b4f43b03e5cedcc9d302a
I20240503 08:23:18.827534 21823 ts_tablet_manager.cc:1398] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:23:18.828827 21838 tablet_copy_client.cc:308] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:23:18.829957 28524 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet 745d736d43244d4f9063f61de77cb473 from peer 4e4a1242f3714d03a6c9ee51dd63929c ({username='slave'} at 127.20.107.129:46737)
I20240503 08:23:18.830111 28524 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet 745d736d43244d4f9063f61de77cb473 from peer 4e4a1242f3714d03a6c9ee51dd63929c at {username='slave'} at 127.20.107.129:46737: session id = 4e4a1242f3714d03a6c9ee51dd63929c-745d736d43244d4f9063f61de77cb473
I20240503 08:23:18.830096 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session 93f7e733cec9415eb422b0a7ba8621cb-d60a162155504975a39c20eff88341e1 received from {username='slave'} at 127.20.107.132:55517
I20240503 08:23:18.830247 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session 93f7e733cec9415eb422b0a7ba8621cb-d60a162155504975a39c20eff88341e1 on tablet d60a162155504975a39c20eff88341e1 with peer 93f7e733cec9415eb422b0a7ba8621cb
I20240503 08:23:18.832480 28524 tablet_copy_source_session.cc:205] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:18.833571 21838 ts_tablet_manager.cc:1875] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:23:18.840790 21838 ts_tablet_manager.cc:1888] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:23:18.841022 21902 ts_tablet_manager.cc:902] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:23:18.841113 21838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745d736d43244d4f9063f61de77cb473. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:18.842672 21905 ts_tablet_manager.cc:902] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:23:18.845305 21838 tablet_copy_client.cc:783] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 5 data blocks...
W20240503 08:23:18.848754 28217 catalog_manager.cc:4573] TS c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0caf86850b3841ea811b0a6ab19ae330: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
I20240503 08:23:18.850387 21902 tablet_copy_client.cc:308] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:23:18.851243 28524 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet a8b4e398d34443b4a440f9ba48555d84 from peer 4e4a1242f3714d03a6c9ee51dd63929c ({username='slave'} at 127.20.107.129:46737)
I20240503 08:23:18.851404 28524 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet a8b4e398d34443b4a440f9ba48555d84 from peer 4e4a1242f3714d03a6c9ee51dd63929c at {username='slave'} at 127.20.107.129:46737: session id = 4e4a1242f3714d03a6c9ee51dd63929c-a8b4e398d34443b4a440f9ba48555d84
I20240503 08:23:18.852003 21905 tablet_copy_client.cc:308] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:23:18.852553 28523 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet 320d1e880958427a88e93321171cb36e from peer 4e4a1242f3714d03a6c9ee51dd63929c ({username='slave'} at 127.20.107.129:46737)
I20240503 08:23:18.852711 28523 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet 320d1e880958427a88e93321171cb36e from peer 4e4a1242f3714d03a6c9ee51dd63929c at {username='slave'} at 127.20.107.129:46737: session id = 4e4a1242f3714d03a6c9ee51dd63929c-320d1e880958427a88e93321171cb36e
I20240503 08:23:18.854597 21838 tablet_copy_client.cc:647] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:18.854667 28524 tablet_copy_source_session.cc:205] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:18.855441 28523 tablet_copy_source_session.cc:205] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:18.855892 21902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b4e398d34443b4a440f9ba48555d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:18.857836 21838 tablet_copy_client.cc:523] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:18.857885 21905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320d1e880958427a88e93321171cb36e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:18.861476 21838 tablet_bootstrap.cc:492] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:23:18.862720 21902 tablet_copy_client.cc:783] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:18.863095 21905 tablet_copy_client.cc:783] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:18.869053 21902 tablet_copy_client.cc:647] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:18.870819 21902 tablet_copy_client.cc:523] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:18.871587 21905 tablet_copy_client.cc:647] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:18.873754 21905 tablet_copy_client.cc:523] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:18.874258 21902 tablet_bootstrap.cc:492] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:23:18.877350 21905 tablet_bootstrap.cc:492] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:23:18.879207 21838 tablet_bootstrap.cc:492] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:18.879879 21838 tablet_bootstrap.cc:492] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap complete.
I20240503 08:23:18.880127 21838 ts_tablet_manager.cc:1367] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20240503 08:23:18.881106 21838 raft_consensus.cc:348] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:18.881275 21838 raft_consensus.cc:729] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: LEARNER
I20240503 08:23:18.881456 21838 consensus_queue.cc:260] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:18.882004 21838 ts_tablet_manager.cc:1398] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:23:18.883016 28523 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-745d736d43244d4f9063f61de77cb473 received from {username='slave'} at 127.20.107.129:46737
I20240503 08:23:18.883177 28523 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-745d736d43244d4f9063f61de77cb473 on tablet 745d736d43244d4f9063f61de77cb473 with peer 4e4a1242f3714d03a6c9ee51dd63929c
I20240503 08:23:18.895562 21902 tablet_bootstrap.cc:492] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:18.895958 21902 tablet_bootstrap.cc:492] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap complete.
I20240503 08:23:18.896211 21902 ts_tablet_manager.cc:1367] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.000s
I20240503 08:23:18.896909 21902 raft_consensus.cc:348] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:18.897166 21902 raft_consensus.cc:729] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: LEARNER
I20240503 08:23:18.897361 21902 consensus_queue.cc:260] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:18.897900 21902 ts_tablet_manager.cc:1398] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:23:18.898000 21905 tablet_bootstrap.cc:492] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:18.898344 21905 tablet_bootstrap.cc:492] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap complete.
I20240503 08:23:18.898661 21905 ts_tablet_manager.cc:1367] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.000s
I20240503 08:23:18.898828 28523 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-a8b4e398d34443b4a440f9ba48555d84 received from {username='slave'} at 127.20.107.129:46737
I20240503 08:23:18.898954 28523 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-a8b4e398d34443b4a440f9ba48555d84 on tablet a8b4e398d34443b4a440f9ba48555d84 with peer 4e4a1242f3714d03a6c9ee51dd63929c
I20240503 08:23:18.899276 21905 raft_consensus.cc:348] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:18.899520 21905 raft_consensus.cc:729] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: LEARNER
I20240503 08:23:18.899677 21905 consensus_queue.cc:260] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:18.900207 21905 ts_tablet_manager.cc:1398] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:23:18.900794 28523 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-320d1e880958427a88e93321171cb36e received from {username='slave'} at 127.20.107.129:46737
I20240503 08:23:18.900905 28523 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-320d1e880958427a88e93321171cb36e on tablet 320d1e880958427a88e93321171cb36e with peer 4e4a1242f3714d03a6c9ee51dd63929c
W20240503 08:23:18.902208 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:18.980619 28373 raft_consensus.cc:1203] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:23:18.995227 21905 ts_tablet_manager.cc:902] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:23:18.996359 21905 tablet_copy_client.cc:308] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:23:18.997048 28523 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet da529c8f8de241d38678d1596b9402ac from peer 4e4a1242f3714d03a6c9ee51dd63929c ({username='slave'} at 127.20.107.129:46737)
I20240503 08:23:18.997231 28523 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet da529c8f8de241d38678d1596b9402ac from peer 4e4a1242f3714d03a6c9ee51dd63929c at {username='slave'} at 127.20.107.129:46737: session id = 4e4a1242f3714d03a6c9ee51dd63929c-da529c8f8de241d38678d1596b9402ac
I20240503 08:23:19.000425 28523 tablet_copy_source_session.cc:205] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 5 blocks and 1 log segments
W20240503 08:23:19.001641 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:23:19.002115 21905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da529c8f8de241d38678d1596b9402ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:19.006492 21905 tablet_copy_client.cc:783] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:19.012208 21905 tablet_copy_client.cc:647] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:19.013808 21905 tablet_copy_client.cc:523] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:19.016621 21905 tablet_bootstrap.cc:492] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:23:19.029529 21905 tablet_bootstrap.cc:492] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:19.029946 21905 tablet_bootstrap.cc:492] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap complete.
I20240503 08:23:19.030164 21905 ts_tablet_manager.cc:1367] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.014s user 0.015s sys 0.000s
I20240503 08:23:19.030808 21905 raft_consensus.cc:348] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:19.030990 21905 raft_consensus.cc:729] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: LEARNER
I20240503 08:23:19.031200 21905 consensus_queue.cc:260] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:19.031751 21905 ts_tablet_manager.cc:1398] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:23:19.032507 28523 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-da529c8f8de241d38678d1596b9402ac received from {username='slave'} at 127.20.107.129:46737
I20240503 08:23:19.032650 28523 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-da529c8f8de241d38678d1596b9402ac on tablet da529c8f8de241d38678d1596b9402ac with peer 4e4a1242f3714d03a6c9ee51dd63929c
I20240503 08:23:19.042315 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(a8b4e398d34443b4a440f9ba48555d84): 4838 bytes on disk
I20240503 08:23:19.042855 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(564fb1b51abd4f8da0372ec99e3c599c): 4581 bytes on disk
I20240503 08:23:19.042927 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(a8b4e398d34443b4a440f9ba48555d84) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240503 08:23:19.043267 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(d60a162155504975a39c20eff88341e1): 4921 bytes on disk
I20240503 08:23:19.043452 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(564fb1b51abd4f8da0372ec99e3c599c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240503 08:23:19.043874 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(d60a162155504975a39c20eff88341e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240503 08:23:19.044310 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(320d1e880958427a88e93321171cb36e): 4583 bytes on disk
I20240503 08:23:19.045002 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(320d1e880958427a88e93321171cb36e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240503 08:23:19.045958 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(745d736d43244d4f9063f61de77cb473): 4583 bytes on disk
I20240503 08:23:19.046402 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(745d736d43244d4f9063f61de77cb473) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 08:23:19.047248 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(da529c8f8de241d38678d1596b9402ac): 4583 bytes on disk
I20240503 08:23:19.047678 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(da529c8f8de241d38678d1596b9402ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 08:23:19.065128 21905 ts_tablet_manager.cc:902] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:23:19.066848 21905 tablet_copy_client.cc:308] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:23:19.067473 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 0caf86850b3841ea811b0a6ab19ae330 from peer 4e4a1242f3714d03a6c9ee51dd63929c ({username='slave'} at 127.20.107.129:57301)
I20240503 08:23:19.067600 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 0caf86850b3841ea811b0a6ab19ae330 from peer 4e4a1242f3714d03a6c9ee51dd63929c at {username='slave'} at 127.20.107.129:57301: session id = 4e4a1242f3714d03a6c9ee51dd63929c-0caf86850b3841ea811b0a6ab19ae330
I20240503 08:23:19.069890 28785 tablet_copy_source_session.cc:205] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:19.070058 21892 ts_tablet_manager.cc:902] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:23:19.071532 21905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caf86850b3841ea811b0a6ab19ae330. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:19.073348 21892 tablet_copy_client.cc:308] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:23:19.074038 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet a8a9adc3712d436c95505a3216e7607c from peer 716ffbb2a00b4f43b03e5cedcc9d302a ({username='slave'} at 127.20.107.130:45037)
I20240503 08:23:19.074173 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet a8a9adc3712d436c95505a3216e7607c from peer 716ffbb2a00b4f43b03e5cedcc9d302a at {username='slave'} at 127.20.107.130:45037: session id = 716ffbb2a00b4f43b03e5cedcc9d302a-a8a9adc3712d436c95505a3216e7607c
I20240503 08:23:19.077255 28394 tablet_copy_source_session.cc:205] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:19.078250 21905 tablet_copy_client.cc:783] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:19.078258 21892 ts_tablet_manager.cc:1875] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:23:19.083901 21892 ts_tablet_manager.cc:1888] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:23:19.084168 21892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a9adc3712d436c95505a3216e7607c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:19.084246 21905 tablet_copy_client.cc:647] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:19.086313 21905 tablet_copy_client.cc:523] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:19.087085 21892 tablet_copy_client.cc:783] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Starting download of 5 data blocks...
W20240503 08:23:19.089349 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:19.090096 21905 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:23:19.093458 21892 tablet_copy_client.cc:647] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:19.095052 21892 tablet_copy_client.cc:523] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:19.097486 21892 tablet_bootstrap.cc:492] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:23:19.107792 21905 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:19.108104 21905 tablet_bootstrap.cc:492] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap complete.
I20240503 08:23:19.108305 21905 ts_tablet_manager.cc:1367] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20240503 08:23:19.108815 21905 raft_consensus.cc:348] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:19.109025 21905 raft_consensus.cc:729] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: LEARNER
I20240503 08:23:19.109189 21905 consensus_queue.cc:260] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:19.109791 21905 ts_tablet_manager.cc:1398] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240503 08:23:19.109925 21892 tablet_bootstrap.cc:492] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:19.110275 21892 tablet_bootstrap.cc:492] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap complete.
I20240503 08:23:19.110522 21892 ts_tablet_manager.cc:1367] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.002s
I20240503 08:23:19.110523 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-0caf86850b3841ea811b0a6ab19ae330 received from {username='slave'} at 127.20.107.129:57301
I20240503 08:23:19.110644 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-0caf86850b3841ea811b0a6ab19ae330 on tablet 0caf86850b3841ea811b0a6ab19ae330 with peer 4e4a1242f3714d03a6c9ee51dd63929c
I20240503 08:23:19.111155 21892 raft_consensus.cc:348] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:23:19.111333 21892 raft_consensus.cc:729] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: LEARNER
I20240503 08:23:19.111514 21892 consensus_queue.cc:260] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:23:19.114025 21892 ts_tablet_manager.cc:1398] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:23:19.114626 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session 716ffbb2a00b4f43b03e5cedcc9d302a-a8a9adc3712d436c95505a3216e7607c received from {username='slave'} at 127.20.107.130:45037
I20240503 08:23:19.114719 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session 716ffbb2a00b4f43b03e5cedcc9d302a-a8a9adc3712d436c95505a3216e7607c on tablet a8a9adc3712d436c95505a3216e7607c with peer 716ffbb2a00b4f43b03e5cedcc9d302a
W20240503 08:23:19.146090 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:23:19.173480 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:19.232745 21782 consensus_queue.cc:579] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Leader has been unable to successfully communicate with peer b29361eb4afa4ab3af7d8019d8a5042d for more than 300 seconds (300.354s)
W20240503 08:23:19.233234 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:19.237279 28631 consensus_queue.cc:237] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } }
I20240503 08:23:19.239326 28762 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:23:19.239356 28502 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:23:19.240016 21965 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:23:19.240253 28373 raft_consensus.cc:1203] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15] Dedup: 1.15->[]
I20240503 08:23:19.240253 21782 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:23:19.243187 28501 raft_consensus.cc:1203] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:23:19.244778 21964 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:19.247790 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 09156806b0744752969b6a574e24b50a with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:19.248064 28502 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:19.249276 28232 catalog_manager.cc:5484] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 10 to 11, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:23:19.257207 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:23:19.259838 28762 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:19.266067 28631 consensus_queue.cc:237] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
W20240503 08:23:19.269527 28569 consensus_peers.cc:485] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d -> Peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): Couldn't send request to peer 4e4a1242f3714d03a6c9ee51dd63929c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:19.269613 28502 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:23:19.269631 28762 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:23:19.270143 21963 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:23:19.270474 21965 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:23:19.272651 21782 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:19.273327 28762 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:19.273420 28502 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:19.279289 28217 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 09156806b0744752969b6a574e24b50a with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:19.279207 28232 catalog_manager.cc:5484] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 11 to 12, NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } } }
I20240503 08:23:19.295928 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(a8a9adc3712d436c95505a3216e7607c): 4583 bytes on disk
I20240503 08:23:19.296545 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(a8a9adc3712d436c95505a3216e7607c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 08:23:19.299811 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(0caf86850b3841ea811b0a6ab19ae330): 4583 bytes on disk
I20240503 08:23:19.300527 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(0caf86850b3841ea811b0a6ab19ae330) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
I20240503 08:23:19.329959 28373 raft_consensus.cc:1203] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240503 08:23:19.342588 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:23:19.353931 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:19.355302 21976 raft_consensus.cc:1050] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb to VOTER
I20240503 08:23:19.355950 21976 consensus_queue.cc:237] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } }
I20240503 08:23:19.357722 28762 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:23:19.358300 28502 raft_consensus.cc:1261] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 08:23:19.358575 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:19.358588 21847 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:23:19.359015 21976 consensus_queue.cc:1035] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:23:19.361718 21847 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:19.362313 28762 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:19.366513 28373 raft_consensus.cc:1203] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:23:19.366803 28761 raft_consensus.cc:1203] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 08:23:19.372335 28232 catalog_manager.cc:5484] T 173376257c4946b5a801a27e4dffe90d P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 11 to 12, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:19.373909 28502 raft_consensus.cc:2932] T 173376257c4946b5a801a27e4dffe90d P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:19.376158 20903 raft_consensus.cc:1050] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c to VOTER
I20240503 08:23:19.377007 20903 consensus_queue.cc:237] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } }
I20240503 08:23:19.380004 28373 raft_consensus.cc:1203] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:23:19.382957 28631 raft_consensus.cc:1261] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:23:19.383673 21754 consensus_queue.cc:1035] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:23:19.385223 28698 consensus_peers.cc:485] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:19.389173 28373 raft_consensus.cc:1261] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:23:19.389648 21754 consensus_queue.cc:1035] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:23:19.391896 21801 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.392489 28373 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.397091 28631 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.398012 28228 catalog_manager.cc:5484] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:19.404093 28762 consensus_queue.cc:237] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } }
I20240503 08:23:19.407066 28373 raft_consensus.cc:1261] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:23:19.407232 28631 raft_consensus.cc:1261] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:23:19.407693 20903 consensus_queue.cc:1035] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:19.408080 20903 consensus_queue.cc:1035] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:19.409952 21847 consensus_queue.cc:579] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Leader has been unable to successfully communicate with peer b29361eb4afa4ab3af7d8019d8a5042d for more than 300 seconds (300.389s)
I20240503 08:23:19.410285 20903 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.411679 28631 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.413913 28374 raft_consensus.cc:2932] T 6e9537f30aaf45adbc2cc4a784009de7 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.414973 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e9537f30aaf45adbc2cc4a784009de7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:19.415891 28226 catalog_manager.cc:5484] T 6e9537f30aaf45adbc2cc4a784009de7 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:19.418624 28373 consensus_queue.cc:237] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
W20240503 08:23:19.425069 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:19.426060 28631 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 08:23:19.426702 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:19.426748 21980 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:23:19.434031 21847 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:23:19.436213 28631 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:23:19.438796 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d13c33b8712e4c488abb99b73ff42b86 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:19.442060 28226 catalog_manager.cc:5484] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 10 to 11, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:19.447352 28373 consensus_queue.cc:237] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } }
I20240503 08:23:19.449525 28631 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 08:23:19.450240 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095): Couldn't send request to peer 93f7e733cec9415eb422b0a7ba8621cb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:19.450253 21980 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 08:23:19.450788 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:19.452324 21847 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } } }
I20240503 08:23:19.452924 28631 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } } }
I20240503 08:23:19.456960 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d13c33b8712e4c488abb99b73ff42b86 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:19.458065 28226 catalog_manager.cc:5484] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 11 to 12, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:19.553449 21972 consensus_queue.cc:579] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Leader has been unable to successfully communicate with peer b29361eb4afa4ab3af7d8019d8a5042d for more than 300 seconds (300.302s)
I20240503 08:23:19.572669 21974 consensus_queue.cc:579] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Leader has been unable to successfully communicate with peer b29361eb4afa4ab3af7d8019d8a5042d for more than 300 seconds (300.017s)
I20240503 08:23:19.576448 28502 consensus_queue.cc:237] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:23:19.578524 28762 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:23:19.579080 21975 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 08:23:19.579464 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:19.581374 21974 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:23:19.581925 28762 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:23:19.584211 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b5736ee8065344ea8124fdf759035216 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:19.586264 28228 catalog_manager.cc:5484] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 12 to 13, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } }
I20240503 08:23:19.590999 28502 consensus_queue.cc:237] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
W20240503 08:23:19.593043 28438 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): Couldn't send request to peer 4574396f35514c9caccb346f1a7af84d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:19.593138 28762 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:23:19.593825 21974 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 08:23:19.594071 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:19.595764 21582 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:23:19.596297 28762 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:23:19.598433 28373 raft_consensus.cc:1203] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:23:19.601893 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b5736ee8065344ea8124fdf759035216 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:19.602946 28228 catalog_manager.cc:5484] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 13 to 14, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:23:19.629166 28373 raft_consensus.cc:1203] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15] Dedup: 1.15->[]
I20240503 08:23:19.658233 21801 raft_consensus.cc:1050] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a to VOTER
I20240503 08:23:19.659113 21801 consensus_queue.cc:237] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } }
I20240503 08:23:19.673980 28502 raft_consensus.cc:1203] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:23:19.676306 28373 raft_consensus.cc:1261] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:23:19.677178 21754 consensus_queue.cc:1035] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:23:19.677549 28502 raft_consensus.cc:1261] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:23:19.679852 21754 consensus_queue.cc:1035] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:23:19.680550 28698 consensus_peers.cc:485] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:23:19.682796 28308 consensus_peers.cc:462] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:23:19.684746 21782 consensus_queue.cc:579] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Leader has been unable to successfully communicate with peer bea2145b8e24436392b18ec3cfe90b93 for more than 300 seconds (300.434s)
W20240503 08:23:19.686517 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:19.696010 21801 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:19.699855 22013 ts_tablet_manager.cc:902] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:23:19.699877 28373 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:19.705780 28502 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:19.706701 28232 catalog_manager.cc:5484] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 7 to 8, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:19.720275 28762 consensus_queue.cc:237] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } }
I20240503 08:23:19.724190 28373 raft_consensus.cc:1261] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:23:19.724368 28502 raft_consensus.cc:1261] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:23:19.724973 22010 consensus_queue.cc:1035] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:19.727849 21801 consensus_queue.cc:1035] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:19.729120 22010 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:19.729737 28373 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:19.732188 28502 raft_consensus.cc:2932] T 564fb1b51abd4f8da0372ec99e3c599c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:19.737924 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 564fb1b51abd4f8da0372ec99e3c599c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:19.741487 28232 catalog_manager.cc:5484] T 564fb1b51abd4f8da0372ec99e3c599c P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:19.745744 22013 tablet_copy_client.cc:308] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:23:19.746912 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet 09156806b0744752969b6a574e24b50a from peer 4e4a1242f3714d03a6c9ee51dd63929c ({username='slave'} at 127.20.107.129:40871)
I20240503 08:23:19.747090 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet 09156806b0744752969b6a574e24b50a from peer 4e4a1242f3714d03a6c9ee51dd63929c at {username='slave'} at 127.20.107.129:40871: session id = 4e4a1242f3714d03a6c9ee51dd63929c-09156806b0744752969b6a574e24b50a
I20240503 08:23:19.750298 28655 tablet_copy_source_session.cc:205] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:19.751608 22013 ts_tablet_manager.cc:1875] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240503 08:23:19.756299 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 08:23:19.759018 22013 ts_tablet_manager.cc:1888] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:23:19.759325 22013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09156806b0744752969b6a574e24b50a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:19.761996 22013 tablet_copy_client.cc:783] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 5 data blocks...
W20240503 08:23:19.763859 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:23:19.769088 22013 tablet_copy_client.cc:647] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:19.770857 22013 tablet_copy_client.cc:523] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:19.773631 22013 tablet_bootstrap.cc:492] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap starting.
I20240503 08:23:19.789307 22013 tablet_bootstrap.cc:492] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:19.789644 22013 tablet_bootstrap.cc:492] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: Bootstrap complete.
I20240503 08:23:19.789844 22013 ts_tablet_manager.cc:1367] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.001s
I20240503 08:23:19.790407 22013 raft_consensus.cc:348] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:19.790613 22013 raft_consensus.cc:729] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e4a1242f3714d03a6c9ee51dd63929c, State: Initialized, Role: LEARNER
I20240503 08:23:19.790832 22013 consensus_queue.cc:260] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: NON_VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: true } }
I20240503 08:23:19.791294 22013 ts_tablet_manager.cc:1398] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:23:19.792026 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-09156806b0744752969b6a574e24b50a received from {username='slave'} at 127.20.107.129:40871
I20240503 08:23:19.792143 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session 4e4a1242f3714d03a6c9ee51dd63929c-09156806b0744752969b6a574e24b50a on tablet 09156806b0744752969b6a574e24b50a with peer 4e4a1242f3714d03a6c9ee51dd63929c
I20240503 08:23:19.798110 22003 raft_consensus.cc:1050] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c to VOTER
I20240503 08:23:19.798879 22003 consensus_queue.cc:237] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } }
I20240503 08:23:19.800902 28373 raft_consensus.cc:1261] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:23:19.801394 28631 raft_consensus.cc:1261] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:23:19.801421 21582 consensus_queue.cc:1035] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:23:19.801837 28438 consensus_peers.cc:485] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:19.802119 21582 consensus_queue.cc:1035] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:23:19.804121 28420 maintenance_manager.cc:419] P 4e4a1242f3714d03a6c9ee51dd63929c: Scheduling UndoDeltaBlockGCOp(09156806b0744752969b6a574e24b50a): 4838 bytes on disk
I20240503 08:23:19.804646 28311 maintenance_manager.cc:643] P 4e4a1242f3714d03a6c9ee51dd63929c: UndoDeltaBlockGCOp(09156806b0744752969b6a574e24b50a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240503 08:23:19.806607 21975 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.807328 28631 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.811120 28373 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.812801 28232 catalog_manager.cc:5484] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:19.817421 28502 consensus_queue.cc:237] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } }
I20240503 08:23:19.819003 28631 raft_consensus.cc:1261] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:23:19.819710 21975 consensus_queue.cc:1035] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:19.821663 28374 raft_consensus.cc:1261] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:23:19.822100 21754 raft_consensus.cc:1050] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c to VOTER
I20240503 08:23:19.822851 21754 consensus_queue.cc:237] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } }
I20240503 08:23:19.824330 21975 consensus_queue.cc:1035] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:19.825222 22003 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.826130 28373 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240503 08:23:19.826121 28631 raft_consensus.cc:1261] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240503 08:23:19.826578 22010 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240503 08:23:19.826800 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:19.826983 22010 consensus_queue.cc:1035] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 08:23:19.828807 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 745d736d43244d4f9063f61de77cb473 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:19.829547 28374 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.829816 28635 raft_consensus.cc:2932] T 745d736d43244d4f9063f61de77cb473 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.831048 28232 catalog_manager.cc:5484] T 745d736d43244d4f9063f61de77cb473 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:23:19.838652 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:19.840306 21975 raft_consensus.cc:1050] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c to VOTER
I20240503 08:23:19.841089 21975 consensus_queue.cc:237] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } }
I20240503 08:23:19.841473 22010 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.842144 28631 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.843796 28762 raft_consensus.cc:1261] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:23:19.844089 28374 raft_consensus.cc:1261] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:23:19.845055 22003 consensus_queue.cc:1035] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:23:19.845530 28439 consensus_peers.cc:485] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:19.847743 22003 consensus_queue.cc:1035] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:23:19.848218 28374 raft_consensus.cc:2932] T c6e1b9f104ea471cab7527bd56433327 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.849138 28228 catalog_manager.cc:5484] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 15 to 16, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:19.858539 21974 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.859081 28762 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.862501 28374 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.863883 28228 catalog_manager.cc:5484] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:19.869977 28501 consensus_queue.cc:237] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } }
I20240503 08:23:19.871783 28762 raft_consensus.cc:1261] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:23:19.872277 21974 consensus_queue.cc:1035] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:19.872769 28374 raft_consensus.cc:1261] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:23:19.873270 21974 consensus_queue.cc:1035] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:23:19.873962 28217 catalog_manager.cc:4573] TS c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 745d736d43244d4f9063f61de77cb473: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
I20240503 08:23:19.874819 21582 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.875351 28762 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.876691 28374 raft_consensus.cc:2932] T 320d1e880958427a88e93321171cb36e P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.878813 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 320d1e880958427a88e93321171cb36e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:19.879671 28232 catalog_manager.cc:5484] T 320d1e880958427a88e93321171cb36e P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 8 to 9, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:19.894436 22038 ts_tablet_manager.cc:902] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:23:19.895825 22038 tablet_copy_client.cc:308] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:23:19.896467 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet d13c33b8712e4c488abb99b73ff42b86 from peer 93f7e733cec9415eb422b0a7ba8621cb ({username='slave'} at 127.20.107.132:50515)
I20240503 08:23:19.896607 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet d13c33b8712e4c488abb99b73ff42b86 from peer 93f7e733cec9415eb422b0a7ba8621cb at {username='slave'} at 127.20.107.132:50515: session id = 93f7e733cec9415eb422b0a7ba8621cb-d13c33b8712e4c488abb99b73ff42b86
I20240503 08:23:19.898774 28394 tablet_copy_source_session.cc:205] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:19.899758 22038 ts_tablet_manager.cc:1875] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:23:19.904739 22038 ts_tablet_manager.cc:1888] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:23:19.905023 22038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d13c33b8712e4c488abb99b73ff42b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:19.907059 22038 tablet_copy_client.cc:783] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:19.912441 22038 tablet_copy_client.cc:647] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:19.913998 22038 tablet_copy_client.cc:523] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:19.916276 22038 tablet_bootstrap.cc:492] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:23:19.927528 21972 raft_consensus.cc:1050] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb to VOTER
I20240503 08:23:19.928545 21972 consensus_queue.cc:237] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } }
I20240503 08:23:19.931239 28762 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 08:23:19.932104 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:19.935366 22038 tablet_bootstrap.cc:492] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:19.935910 22038 tablet_bootstrap.cc:492] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap complete.
I20240503 08:23:19.936239 22038 ts_tablet_manager.cc:1367] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20240503 08:23:19.937263 22038 raft_consensus.cc:348] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } }
I20240503 08:23:19.937613 22038 raft_consensus.cc:729] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: LEARNER
I20240503 08:23:19.937868 22038 consensus_queue.cc:260] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } }
I20240503 08:23:19.938668 22038 ts_tablet_manager.cc:1398] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 08:23:19.939579 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session 93f7e733cec9415eb422b0a7ba8621cb-d13c33b8712e4c488abb99b73ff42b86 received from {username='slave'} at 127.20.107.132:50515
I20240503 08:23:19.939766 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session 93f7e733cec9415eb422b0a7ba8621cb-d13c33b8712e4c488abb99b73ff42b86 on tablet d13c33b8712e4c488abb99b73ff42b86 with peer 93f7e733cec9415eb422b0a7ba8621cb
I20240503 08:23:19.946552 22045 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:23:19.946888 28374 raft_consensus.cc:1261] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 08:23:19.947362 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:23:19.947765 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:19.947762 22045 consensus_queue.cc:1035] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:23:19.950203 21972 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:19.950841 28374 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:19.954119 28762 raft_consensus.cc:2932] T d60a162155504975a39c20eff88341e1 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:19.955440 28228 catalog_manager.cc:5484] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 10 to 11, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: NON_VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:19.982069 21974 raft_consensus.cc:1050] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c to VOTER
I20240503 08:23:19.982793 21974 consensus_queue.cc:237] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } }
W20240503 08:23:19.986212 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 08:23:19.986200 28374 raft_consensus.cc:1261] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:23:19.986272 28762 raft_consensus.cc:1261] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:23:19.986905 21975 consensus_queue.cc:1035] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:23:19.987687 22003 consensus_queue.cc:1035] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:23:19.988727 28438 consensus_peers.cc:485] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:19.990550 22003 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.991360 28762 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.995378 28374 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:19.997280 28232 catalog_manager.cc:5484] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:23:20.002229 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 08:23:20.003242 28501 consensus_queue.cc:237] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } }
I20240503 08:23:20.004943 28374 raft_consensus.cc:1261] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:23:20.005225 28762 raft_consensus.cc:1261] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:23:20.006441 22003 consensus_queue.cc:1035] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:20.007054 21974 consensus_queue.cc:1035] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:20.009383 21975 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:20.010267 28762 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:20.010246 28374 raft_consensus.cc:2932] T a8b4e398d34443b4a440f9ba48555d84 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:20.013976 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a8b4e398d34443b4a440f9ba48555d84 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:20.015758 28228 catalog_manager.cc:5484] T a8b4e398d34443b4a440f9ba48555d84 P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:20.049373 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(d13c33b8712e4c488abb99b73ff42b86): 4583 bytes on disk
I20240503 08:23:20.050004 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(d13c33b8712e4c488abb99b73ff42b86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240503 08:23:20.053189 22067 ts_tablet_manager.cc:902] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: Initiating tablet copy from peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397)
I20240503 08:23:20.055645 22067 tablet_copy_client.cc:308] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.130:40397
I20240503 08:23:20.056355 28523 tablet_copy_service.cc:140] P 716ffbb2a00b4f43b03e5cedcc9d302a: Received BeginTabletCopySession request for tablet b5736ee8065344ea8124fdf759035216 from peer 4574396f35514c9caccb346f1a7af84d ({username='slave'} at 127.20.107.131:48771)
I20240503 08:23:20.056500 28523 tablet_copy_service.cc:161] P 716ffbb2a00b4f43b03e5cedcc9d302a: Beginning new tablet copy session on tablet b5736ee8065344ea8124fdf759035216 from peer 4574396f35514c9caccb346f1a7af84d at {username='slave'} at 127.20.107.131:48771: session id = 4574396f35514c9caccb346f1a7af84d-b5736ee8065344ea8124fdf759035216
I20240503 08:23:20.058676 28523 tablet_copy_source_session.cc:205] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:20.059726 22067 ts_tablet_manager.cc:1875] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:23:20.062959 21980 raft_consensus.cc:1050] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a to VOTER
I20240503 08:23:20.063926 21980 consensus_queue.cc:237] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } }
I20240503 08:23:20.066368 28501 raft_consensus.cc:1261] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:23:20.066483 28762 raft_consensus.cc:1261] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:23:20.067010 21847 consensus_queue.cc:1035] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:23:20.067473 21847 consensus_queue.cc:1035] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:23:20.067884 28307 consensus_peers.cc:485] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:20.068941 22067 ts_tablet_manager.cc:1888] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240503 08:23:20.069237 22067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5736ee8065344ea8124fdf759035216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:20.077203 22067 tablet_copy_client.cc:783] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:20.078667 21980 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:20.079377 28374 raft_consensus.cc:1203] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240503 08:23:20.083180 28762 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:20.083887 22067 tablet_copy_client.cc:647] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:20.085880 22067 tablet_copy_client.cc:523] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:20.085631 28501 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:20.087422 28226 catalog_manager.cc:5484] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 7 to 8, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:20.089823 22067 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:23:20.095405 28374 consensus_queue.cc:237] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } }
I20240503 08:23:20.097064 28501 raft_consensus.cc:1261] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:23:20.097149 28762 raft_consensus.cc:1261] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:23:20.097832 21980 consensus_queue.cc:1035] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:20.098326 22073 consensus_queue.cc:1035] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:20.102165 28762 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:20.101306 21980 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:20.107745 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a8a9adc3712d436c95505a3216e7607c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:20.107795 28501 raft_consensus.cc:2932] T a8a9adc3712d436c95505a3216e7607c P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:20.109978 21980 consensus_queue.cc:579] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Leader has been unable to successfully communicate with peer c8375219addc48fa99ac474ed1c9aef9 for more than 300 seconds (300.033s)
I20240503 08:23:20.110657 28232 catalog_manager.cc:5484] T a8a9adc3712d436c95505a3216e7607c P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 8 to 9, VOTER c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:20.113997 22067 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:20.114220 21582 raft_consensus.cc:1050] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c to VOTER
I20240503 08:23:20.114468 22067 tablet_bootstrap.cc:492] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: Bootstrap complete.
I20240503 08:23:20.114758 22067 ts_tablet_manager.cc:1367] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.008s
I20240503 08:23:20.114938 21582 consensus_queue.cc:237] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } }
I20240503 08:23:20.115535 22067 raft_consensus.cc:348] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:23:20.115851 22067 raft_consensus.cc:729] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: LEARNER
I20240503 08:23:20.116078 22067 consensus_queue.cc:260] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:23:20.118012 22067 ts_tablet_manager.cc:1398] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:23:20.118659 28523 tablet_copy_service.cc:342] P 716ffbb2a00b4f43b03e5cedcc9d302a: Request end of tablet copy session 4574396f35514c9caccb346f1a7af84d-b5736ee8065344ea8124fdf759035216 received from {username='slave'} at 127.20.107.131:48771
I20240503 08:23:20.118793 28523 tablet_copy_service.cc:433] P 716ffbb2a00b4f43b03e5cedcc9d302a: ending tablet copy session 4574396f35514c9caccb346f1a7af84d-b5736ee8065344ea8124fdf759035216 on tablet b5736ee8065344ea8124fdf759035216 with peer 4574396f35514c9caccb346f1a7af84d
I20240503 08:23:20.119724 28631 raft_consensus.cc:1261] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:23:20.120357 22003 consensus_queue.cc:1035] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:23:20.120491 28374 raft_consensus.cc:1261] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:23:20.120919 28439 consensus_peers.cc:485] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:20.121445 22003 consensus_queue.cc:1035] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:23:20.125108 28371 consensus_queue.cc:237] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } }
I20240503 08:23:20.126482 21582 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
W20240503 08:23:20.128458 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095): Couldn't send request to peer 93f7e733cec9415eb422b0a7ba8621cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d44335d8be7245b1bb52b2eab9e42c18. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:23:20.129086 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:23:20.128805 28371 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
W20240503 08:23:20.130604 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:20.131222 28631 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:20.132086 28232 catalog_manager.cc:5484] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 7 to 8, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:20.137368 28501 consensus_queue.cc:237] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } }
I20240503 08:23:20.137666 28631 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:23:20.139003 22073 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:23:20.140760 28631 raft_consensus.cc:1261] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 08:23:20.141384 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:20.142115 21582 consensus_queue.cc:1035] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:20.142221 28371 raft_consensus.cc:1261] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:23:20.142724 21582 consensus_queue.cc:1035] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:20.144404 22003 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:20.147380 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet da529c8f8de241d38678d1596b9402ac with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:20.147639 28631 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:20.154299 28232 catalog_manager.cc:5484] T da529c8f8de241d38678d1596b9402ac P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 8 to 9, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:20.154903 28371 raft_consensus.cc:2932] T da529c8f8de241d38678d1596b9402ac P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:20.157250 22073 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } } }
I20240503 08:23:20.161445 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d44335d8be7245b1bb52b2eab9e42c18 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:20.162930 28635 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } } }
I20240503 08:23:20.163592 28232 catalog_manager.cc:5484] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 11 to 12, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New cstate: current_term: 2 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:20.197788 20903 raft_consensus.cc:1050] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c to VOTER
I20240503 08:23:20.198448 20903 consensus_queue.cc:237] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } }
I20240503 08:23:20.200492 28635 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240503 08:23:20.200583 28371 raft_consensus.cc:1261] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240503 08:23:20.200639 28222 maintenance_manager.cc:419] P e8f5acc8744f4dccbfd654a6d7dab5bf: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1966 bytes on disk
I20240503 08:23:20.201100 21754 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 08:23:20.201570 21754 consensus_queue.cc:1035] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240503 08:23:20.202394 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:20.209237 28221 maintenance_manager.cc:643] P e8f5acc8744f4dccbfd654a6d7dab5bf: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.008s user 0.005s sys 0.003s Metrics: {"cfile_init":1,"dirs.queue_time_us":3071,"dirs.run_cpu_time_us":72,"dirs.run_wall_time_us":73,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4,"thread_start_us":2986,"threads_started":1}
I20240503 08:23:20.210959 21801 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:20.211653 28371 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:20.216830 28232 catalog_manager.cc:5484] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 15 to 16, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:20.226843 21972 consensus_queue.cc:579] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Leader has been unable to successfully communicate with peer b29361eb4afa4ab3af7d8019d8a5042d for more than 300 seconds (300.320s)
W20240503 08:23:20.228683 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:20.230650 28631 consensus_queue.cc:237] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } }
I20240503 08:23:20.233103 28501 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 3 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 08:23:20.233614 28371 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 3 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 08:23:20.234239 22045 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:20.234666 22045 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:20.215332 28635 raft_consensus.cc:2932] T 0caf86850b3841ea811b0a6ab19ae330 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:20.246584 21972 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } }
I20240503 08:23:20.249802 28374 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } }
I20240503 08:23:20.251710 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba833ce23c4c4e81bc1a81125a1208c9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:20.254406 28232 catalog_manager.cc:5484] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 8 to 9, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New cstate: current_term: 3 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:20.254407 28501 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, NON_VOTER b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } }
I20240503 08:23:20.260123 28631 consensus_queue.cc:237] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } }
W20240503 08:23:20.266134 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:23:20.268035 28569 consensus_peers.cc:485] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d -> Peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095): Couldn't send request to peer 93f7e733cec9415eb422b0a7ba8621cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba833ce23c4c4e81bc1a81125a1208c9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:20.268802 28502 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 3 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240503 08:23:20.270043 21972 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:23:20.272452 21965 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 9 to 10, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } } }
I20240503 08:23:20.276472 28501 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 9 to 10, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } } }
I20240503 08:23:20.281524 28217 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ba833ce23c4c4e81bc1a81125a1208c9 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:20.282590 28226 catalog_manager.cc:5484] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 9 to 10, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New cstate: current_term: 3 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:20.283787 28762 raft_consensus.cc:1203] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240503 08:23:20.294052 28371 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 3 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240503 08:23:20.295106 22045 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:23:20.296773 28374 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 9 to 10, NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } } }
I20240503 08:23:20.305706 21582 consensus_queue.cc:579] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Leader has been unable to successfully communicate with peer b29361eb4afa4ab3af7d8019d8a5042d for more than 300 seconds (300.394s)
I20240503 08:23:20.305958 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(b5736ee8065344ea8124fdf759035216): 4838 bytes on disk
I20240503 08:23:20.307044 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(b5736ee8065344ea8124fdf759035216) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":163,"lbm_reads_lt_1ms":4}
W20240503 08:23:20.338291 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 08:23:20.381503 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 08:23:20.427383 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 08:23:20.469069 21582 consensus_queue.cc:579] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Leader has been unable to successfully communicate with peer c8375219addc48fa99ac474ed1c9aef9 for more than 300 seconds (300.557s)
I20240503 08:23:20.500627 21582 consensus_queue.cc:579] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Leader has been unable to successfully communicate with peer 21d21a5b596b4d48a17a1fb54a972844 for more than 300 seconds (300.027s)
I20240503 08:23:20.541224 22116 ts_tablet_manager.cc:902] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:23:20.542282 22116 tablet_copy_client.cc:308] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:23:20.542855 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet d44335d8be7245b1bb52b2eab9e42c18 from peer 93f7e733cec9415eb422b0a7ba8621cb ({username='slave'} at 127.20.107.132:50515)
I20240503 08:23:20.542979 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet d44335d8be7245b1bb52b2eab9e42c18 from peer 93f7e733cec9415eb422b0a7ba8621cb at {username='slave'} at 127.20.107.132:50515: session id = 93f7e733cec9415eb422b0a7ba8621cb-d44335d8be7245b1bb52b2eab9e42c18
I20240503 08:23:20.545069 28394 tablet_copy_source_session.cc:205] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:20.546259 22116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44335d8be7245b1bb52b2eab9e42c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:20.550491 22116 tablet_copy_client.cc:783] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:20.554373 28635 raft_consensus.cc:1203] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14] Dedup: 1.14->[]
I20240503 08:23:20.558753 22116 tablet_copy_client.cc:647] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:20.560582 22116 tablet_copy_client.cc:523] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:20.563436 22116 tablet_bootstrap.cc:492] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:23:20.579109 22116 tablet_bootstrap.cc:492] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:20.579427 22116 tablet_bootstrap.cc:492] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap complete.
I20240503 08:23:20.579653 22116 ts_tablet_manager.cc:1367] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20240503 08:23:20.580226 22116 raft_consensus.cc:348] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } }
I20240503 08:23:20.580420 22116 raft_consensus.cc:729] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: LEARNER
I20240503 08:23:20.580605 22116 consensus_queue.cc:260] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } }
I20240503 08:23:20.581459 22116 ts_tablet_manager.cc:1398] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:23:20.582196 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session 93f7e733cec9415eb422b0a7ba8621cb-d44335d8be7245b1bb52b2eab9e42c18 received from {username='slave'} at 127.20.107.132:50515
I20240503 08:23:20.582319 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session 93f7e733cec9415eb422b0a7ba8621cb-d44335d8be7245b1bb52b2eab9e42c18 on tablet d44335d8be7245b1bb52b2eab9e42c18 with peer 93f7e733cec9415eb422b0a7ba8621cb
I20240503 08:23:20.651554 21801 consensus_queue.cc:579] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Leader has been unable to successfully communicate with peer 21d21a5b596b4d48a17a1fb54a972844 for more than 300 seconds (300.356s)
I20240503 08:23:20.654795 22045 consensus_queue.cc:579] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Leader has been unable to successfully communicate with peer 21d21a5b596b4d48a17a1fb54a972844 for more than 300 seconds (300.241s)
I20240503 08:23:20.674747 22045 raft_consensus.cc:1050] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER 4e4a1242f3714d03a6c9ee51dd63929c to VOTER
I20240503 08:23:20.675611 22045 consensus_queue.cc:237] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } }
I20240503 08:23:20.681485 28374 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:23:20.682098 28501 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:23:20.682554 21972 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:23:20.683168 21965 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:23:20.683581 28762 raft_consensus.cc:1261] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:23:20.684418 22045 consensus_queue.cc:1035] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:23:20.685240 22045 consensus_queue.cc:579] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d [LEADER]: Leader has been unable to successfully communicate with peer 21d21a5b596b4d48a17a1fb54a972844 for more than 300 seconds (300.073s)
I20240503 08:23:20.694073 28762 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:20.693437 22045 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:20.694797 22116 ts_tablet_manager.cc:902] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:23:20.694839 28374 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:20.700317 28501 raft_consensus.cc:2932] T 09156806b0744752969b6a574e24b50a P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } } }
I20240503 08:23:20.701058 28232 catalog_manager.cc:5484] T 09156806b0744752969b6a574e24b50a P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 12 to 13, 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:20.712000 22116 tablet_copy_client.cc:308] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:23:20.712630 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet ba833ce23c4c4e81bc1a81125a1208c9 from peer 93f7e733cec9415eb422b0a7ba8621cb ({username='slave'} at 127.20.107.132:55517)
I20240503 08:23:20.712749 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet ba833ce23c4c4e81bc1a81125a1208c9 from peer 93f7e733cec9415eb422b0a7ba8621cb at {username='slave'} at 127.20.107.132:55517: session id = 93f7e733cec9415eb422b0a7ba8621cb-ba833ce23c4c4e81bc1a81125a1208c9
I20240503 08:23:20.715184 28655 tablet_copy_source_session.cc:205] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:20.716293 22116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba833ce23c4c4e81bc1a81125a1208c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:20.720626 22116 tablet_copy_client.cc:783] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:20.726701 22116 tablet_copy_client.cc:647] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:20.728329 22116 tablet_copy_client.cc:523] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:20.731004 22116 tablet_bootstrap.cc:492] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap starting.
I20240503 08:23:20.746417 22116 tablet_bootstrap.cc:492] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:20.746838 22116 tablet_bootstrap.cc:492] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb: Bootstrap complete.
I20240503 08:23:20.747107 22116 ts_tablet_manager.cc:1367] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.004s
I20240503 08:23:20.748098 22116 raft_consensus.cc:348] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } }
I20240503 08:23:20.748358 22116 raft_consensus.cc:729] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 93f7e733cec9415eb422b0a7ba8621cb, State: Initialized, Role: LEARNER
I20240503 08:23:20.748597 22116 consensus_queue.cc:260] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 3.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: NON_VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: true } }
I20240503 08:23:20.749279 22116 ts_tablet_manager.cc:1398] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:23:20.749971 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session 93f7e733cec9415eb422b0a7ba8621cb-ba833ce23c4c4e81bc1a81125a1208c9 received from {username='slave'} at 127.20.107.132:55517
I20240503 08:23:20.750090 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session 93f7e733cec9415eb422b0a7ba8621cb-ba833ce23c4c4e81bc1a81125a1208c9 on tablet ba833ce23c4c4e81bc1a81125a1208c9 with peer 93f7e733cec9415eb422b0a7ba8621cb
I20240503 08:23:20.753844 21847 raft_consensus.cc:1050] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb to VOTER
I20240503 08:23:20.754737 21847 consensus_queue.cc:237] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } }
I20240503 08:23:20.757494 28635 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:23:20.758249 28762 raft_consensus.cc:1261] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:23:20.758421 22073 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 08:23:20.758776 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:20.759336 21980 consensus_queue.cc:579] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Leader has been unable to successfully communicate with peer 21d21a5b596b4d48a17a1fb54a972844 for more than 300 seconds (300.160s)
I20240503 08:23:20.759583 21980 consensus_queue.cc:1035] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:23:20.764530 28374 consensus_queue.cc:237] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:23:20.766798 21980 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:20.767294 28762 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
W20240503 08:23:20.772596 28307 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): Couldn't send request to peer 4574396f35514c9caccb346f1a7af84d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:20.775884 28226 catalog_manager.cc:5484] T d13c33b8712e4c488abb99b73ff42b86 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 12 to 13, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:20.777274 28631 raft_consensus.cc:2932] T d13c33b8712e4c488abb99b73ff42b86 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:20.783614 28501 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:23:20.784286 22073 consensus_queue.cc:579] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Leader has been unable to successfully communicate with peer 21d21a5b596b4d48a17a1fb54a972844 for more than 300 seconds (300.227s)
I20240503 08:23:20.784483 22151 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 08:23:20.785437 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:20.791520 28374 consensus_queue.cc:237] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:23:20.794150 21847 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:23:20.794976 28501 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:23:20.798233 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 48e31346107a453a81921b23b244e7ff with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:20.801553 28226 catalog_manager.cc:5484] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 12 to 13, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:23:20.805282 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(ba833ce23c4c4e81bc1a81125a1208c9): 4923 bytes on disk
I20240503 08:23:20.806033 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(ba833ce23c4c4e81bc1a81125a1208c9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
W20240503 08:23:20.806685 28307 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559): Couldn't send request to peer 4574396f35514c9caccb346f1a7af84d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9997e8da99c4f45a63bc3db521b33ba. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:23:20.806921 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:20.807561 28811 maintenance_manager.cc:419] P 93f7e733cec9415eb422b0a7ba8621cb: Scheduling UndoDeltaBlockGCOp(d44335d8be7245b1bb52b2eab9e42c18): 4838 bytes on disk
I20240503 08:23:20.808836 28702 maintenance_manager.cc:643] P 93f7e733cec9415eb422b0a7ba8621cb: UndoDeltaBlockGCOp(d44335d8be7245b1bb52b2eab9e42c18) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":212,"lbm_reads_lt_1ms":4}
I20240503 08:23:20.811133 28501 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:23:20.811892 22144 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:23:20.815086 22152 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:23:20.815717 28502 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } } }
I20240503 08:23:20.819026 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e9997e8da99c4f45a63bc3db521b33ba with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:20.820958 28231 catalog_manager.cc:5484] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 11 to 12, NON_VOTER 4574396f35514c9caccb346f1a7af84d (127.20.107.131) added. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:23:20.846876 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:23:20.866016 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 08:23:20.878639 28217 catalog_manager.cc:4573] TS c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 564fb1b51abd4f8da0372ec99e3c599c: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
I20240503 08:23:20.948452 28762 raft_consensus.cc:1203] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
W20240503 08:23:21.009568 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:21.010609 20903 consensus_queue.cc:579] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Leader has been unable to successfully communicate with peer 21d21a5b596b4d48a17a1fb54a972844 for more than 300 seconds (300.377s)
I20240503 08:23:21.014501 28762 consensus_queue.cc:237] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:23:21.016686 28631 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 08:23:21.016728 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): Couldn't send request to peer 716ffbb2a00b4f43b03e5cedcc9d302a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:21.017259 20903 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 08:23:21.017573 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:21.019382 21754 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:23:21.019892 28631 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:23:21.022238 28218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4babf0ea9b954a23b1b1c4e403dce0bd with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:21.023849 28231 catalog_manager.cc:5484] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 11 to 12, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:23:21.043610 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 08:23:21.046876 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 08:23:21.047719 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:21.077958 21972 consensus_queue.cc:579] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Leader has been unable to successfully communicate with peer 21d21a5b596b4d48a17a1fb54a972844 for more than 300 seconds (300.513s)
I20240503 08:23:21.081707 28631 consensus_queue.cc:237] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:23:21.084414 21847 consensus_queue.cc:579] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Leader has been unable to successfully communicate with peer 21d21a5b596b4d48a17a1fb54a972844 for more than 300 seconds (300.483s)
I20240503 08:23:21.084479 28374 raft_consensus.cc:1261] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:23:21.084654 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:21.085256 21965 consensus_queue.cc:1035] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:23:21.085494 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130:40397): Couldn't send request to peer 716ffbb2a00b4f43b03e5cedcc9d302a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28e42c3244f949c7b7343bf587894ac5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:21.088126 21972 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:23:21.088757 28374 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } } }
I20240503 08:23:21.091252 28217 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 28e42c3244f949c7b7343bf587894ac5 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:23:21.092749 28226 catalog_manager.cc:5484] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 6 to 7, NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) added. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:21.173557 21582 raft_consensus.cc:1050] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a: attempting to promote NON_VOTER 4574396f35514c9caccb346f1a7af84d to VOTER
I20240503 08:23:21.174377 21582 consensus_queue.cc:237] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } }
I20240503 08:23:21.176365 28631 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240503 08:23:21.177024 22182 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:23:21.177541 28762 raft_consensus.cc:1261] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Refusing update from remote peer 716ffbb2a00b4f43b03e5cedcc9d302a: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240503 08:23:21.178906 22182 consensus_queue.cc:1035] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240503 08:23:21.179384 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:21.181838 22183 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:21.182519 28762 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:21.183900 22186 ts_tablet_manager.cc:902] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:23:21.187500 22186 tablet_copy_client.cc:308] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:23:21.187011 28631 raft_consensus.cc:2932] T b5736ee8065344ea8124fdf759035216 P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:21.188200 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet e9997e8da99c4f45a63bc3db521b33ba from peer 4574396f35514c9caccb346f1a7af84d ({username='slave'} at 127.20.107.131:46685)
I20240503 08:23:21.188349 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet e9997e8da99c4f45a63bc3db521b33ba from peer 4574396f35514c9caccb346f1a7af84d at {username='slave'} at 127.20.107.131:46685: session id = 4574396f35514c9caccb346f1a7af84d-e9997e8da99c4f45a63bc3db521b33ba
I20240503 08:23:21.188503 28231 catalog_manager.cc:5484] T b5736ee8065344ea8124fdf759035216 P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 14 to 15, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:21.191597 28394 tablet_copy_source_session.cc:205] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:21.193593 22186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9997e8da99c4f45a63bc3db521b33ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:21.199788 22186 tablet_copy_client.cc:783] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:21.202982 28762 raft_consensus.cc:1203] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb [term 3 LEARNER]: Deduplicated request from leader. Original: 3.9->[3.10-3.10] Dedup: 3.10->[]
I20240503 08:23:21.208134 22186 tablet_copy_client.cc:647] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:21.209645 22186 tablet_copy_client.cc:523] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:21.212266 22186 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:23:21.229869 22186 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:21.230190 22186 tablet_bootstrap.cc:492] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d: Bootstrap complete.
I20240503 08:23:21.230412 22186 ts_tablet_manager.cc:1367] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.018s user 0.019s sys 0.000s
I20240503 08:23:21.230957 22186 raft_consensus.cc:348] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:23:21.231132 22186 raft_consensus.cc:729] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: LEARNER
I20240503 08:23:21.231302 22186 consensus_queue.cc:260] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:23:21.231889 22186 ts_tablet_manager.cc:1398] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:23:21.232798 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session 4574396f35514c9caccb346f1a7af84d-e9997e8da99c4f45a63bc3db521b33ba received from {username='slave'} at 127.20.107.131:46685
I20240503 08:23:21.232921 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session 4574396f35514c9caccb346f1a7af84d-e9997e8da99c4f45a63bc3db521b33ba on tablet e9997e8da99c4f45a63bc3db521b33ba with peer 4574396f35514c9caccb346f1a7af84d
I20240503 08:23:21.278213 22186 ts_tablet_manager.cc:902] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: Initiating tablet copy from peer 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529)
I20240503 08:23:21.279300 22186 tablet_copy_client.cc:308] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.129:38529
I20240503 08:23:21.279925 28394 tablet_copy_service.cc:140] P 4e4a1242f3714d03a6c9ee51dd63929c: Received BeginTabletCopySession request for tablet 48e31346107a453a81921b23b244e7ff from peer 4574396f35514c9caccb346f1a7af84d ({username='slave'} at 127.20.107.131:46685)
I20240503 08:23:21.280071 28394 tablet_copy_service.cc:161] P 4e4a1242f3714d03a6c9ee51dd63929c: Beginning new tablet copy session on tablet 48e31346107a453a81921b23b244e7ff from peer 4574396f35514c9caccb346f1a7af84d at {username='slave'} at 127.20.107.131:46685: session id = 4574396f35514c9caccb346f1a7af84d-48e31346107a453a81921b23b244e7ff
I20240503 08:23:21.283090 28394 tablet_copy_source_session.cc:205] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:21.284103 22186 ts_tablet_manager.cc:1875] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:23:21.289196 22186 ts_tablet_manager.cc:1888] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240503 08:23:21.289425 22186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e31346107a453a81921b23b244e7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:21.291884 22186 tablet_copy_client.cc:783] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:21.297713 22186 tablet_copy_client.cc:647] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:21.299288 22186 tablet_copy_client.cc:523] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:21.302371 22186 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: Bootstrap starting.
I20240503 08:23:21.313653 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(e9997e8da99c4f45a63bc3db521b33ba): 4923 bytes on disk
I20240503 08:23:21.314172 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(e9997e8da99c4f45a63bc3db521b33ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 08:23:21.327534 22186 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:21.327958 22186 tablet_bootstrap.cc:492] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: Bootstrap complete.
I20240503 08:23:21.328225 22186 ts_tablet_manager.cc:1367] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.000s
I20240503 08:23:21.328931 22186 raft_consensus.cc:348] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:23:21.329200 22186 raft_consensus.cc:729] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4574396f35514c9caccb346f1a7af84d, State: Initialized, Role: LEARNER
I20240503 08:23:21.329406 22186 consensus_queue.cc:260] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: NON_VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: true } }
I20240503 08:23:21.330088 22186 ts_tablet_manager.cc:1398] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 08:23:21.331383 28394 tablet_copy_service.cc:342] P 4e4a1242f3714d03a6c9ee51dd63929c: Request end of tablet copy session 4574396f35514c9caccb346f1a7af84d-48e31346107a453a81921b23b244e7ff received from {username='slave'} at 127.20.107.131:46685
I20240503 08:23:21.331490 28394 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session 4574396f35514c9caccb346f1a7af84d-48e31346107a453a81921b23b244e7ff on tablet 48e31346107a453a81921b23b244e7ff with peer 4574396f35514c9caccb346f1a7af84d
W20240503 08:23:21.445432 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:23:21.455299 22144 raft_consensus.cc:1050] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb to VOTER
I20240503 08:23:21.456050 22144 consensus_queue.cc:237] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } }
I20240503 08:23:21.458823 28762 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 08:23:21.459321 28631 raft_consensus.cc:1261] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 08:23:21.459650 22144 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 08:23:21.460023 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:23:21.460029 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:21.460078 22152 consensus_queue.cc:1035] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:23:21.463136 22152 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:21.463866 28631 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:21.467077 28762 raft_consensus.cc:2932] T d44335d8be7245b1bb52b2eab9e42c18 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:21.470263 28232 catalog_manager.cc:5484] T d44335d8be7245b1bb52b2eab9e42c18 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 12 to 13, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { replace: true } } peers { permanent_uuid: "c8375219addc48fa99ac474ed1c9aef9" member_type: VOTER last_known_addr { host: "127.20.107.133" port: 35121 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: NON_VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: true } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:21.521133 22234 ts_tablet_manager.cc:902] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a: Initiating tablet copy from peer 4574396f35514c9caccb346f1a7af84d (127.20.107.131:44559)
I20240503 08:23:21.522248 22234 tablet_copy_client.cc:308] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.131:44559
I20240503 08:23:21.522934 28655 tablet_copy_service.cc:140] P 4574396f35514c9caccb346f1a7af84d: Received BeginTabletCopySession request for tablet 28e42c3244f949c7b7343bf587894ac5 from peer 716ffbb2a00b4f43b03e5cedcc9d302a ({username='slave'} at 127.20.107.130:41829)
I20240503 08:23:21.523098 28655 tablet_copy_service.cc:161] P 4574396f35514c9caccb346f1a7af84d: Beginning new tablet copy session on tablet 28e42c3244f949c7b7343bf587894ac5 from peer 716ffbb2a00b4f43b03e5cedcc9d302a at {username='slave'} at 127.20.107.130:41829: session id = 716ffbb2a00b4f43b03e5cedcc9d302a-28e42c3244f949c7b7343bf587894ac5
W20240503 08:23:21.524253 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:23:21.525746 28655 tablet_copy_source_session.cc:205] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:21.526923 22234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e42c3244f949c7b7343bf587894ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:21.531154 22234 tablet_copy_client.cc:783] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:21.536271 22234 tablet_copy_client.cc:647] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:21.537781 22234 tablet_copy_client.cc:523] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:21.540753 22234 tablet_bootstrap.cc:492] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:23:21.541348 22239 ts_tablet_manager.cc:902] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: Initiating tablet copy from peer 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132:34095)
I20240503 08:23:21.542933 22239 tablet_copy_client.cc:308] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Beginning tablet copy session from remote peer at address 127.20.107.132:34095
I20240503 08:23:21.543640 28785 tablet_copy_service.cc:140] P 93f7e733cec9415eb422b0a7ba8621cb: Received BeginTabletCopySession request for tablet 4babf0ea9b954a23b1b1c4e403dce0bd from peer 716ffbb2a00b4f43b03e5cedcc9d302a ({username='slave'} at 127.20.107.130:35911)
I20240503 08:23:21.543783 28785 tablet_copy_service.cc:161] P 93f7e733cec9415eb422b0a7ba8621cb: Beginning new tablet copy session on tablet 4babf0ea9b954a23b1b1c4e403dce0bd from peer 716ffbb2a00b4f43b03e5cedcc9d302a at {username='slave'} at 127.20.107.130:35911: session id = 716ffbb2a00b4f43b03e5cedcc9d302a-4babf0ea9b954a23b1b1c4e403dce0bd
I20240503 08:23:21.546270 28785 tablet_copy_source_session.cc:205] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:23:21.547262 22239 ts_tablet_manager.cc:1875] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:23:21.552094 22239 ts_tablet_manager.cc:1888] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240503 08:23:21.552382 22239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4babf0ea9b954a23b1b1c4e403dce0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:23:21.555001 22239 tablet_copy_client.cc:783] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Starting download of 5 data blocks...
I20240503 08:23:21.560775 22239 tablet_copy_client.cc:647] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:23:21.562332 22239 tablet_copy_client.cc:523] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:23:21.566357 28681 maintenance_manager.cc:419] P 4574396f35514c9caccb346f1a7af84d: Scheduling UndoDeltaBlockGCOp(48e31346107a453a81921b23b244e7ff): 4838 bytes on disk
I20240503 08:23:21.566699 22239 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap starting.
I20240503 08:23:21.567037 28572 maintenance_manager.cc:643] P 4574396f35514c9caccb346f1a7af84d: UndoDeltaBlockGCOp(48e31346107a453a81921b23b244e7ff) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
I20240503 08:23:21.571784 22234 tablet_bootstrap.cc:492] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:21.572116 22234 tablet_bootstrap.cc:492] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap complete.
I20240503 08:23:21.572384 22234 ts_tablet_manager.cc:1367] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.004s
I20240503 08:23:21.573011 22234 raft_consensus.cc:348] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:23:21.573221 22234 raft_consensus.cc:729] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: LEARNER
I20240503 08:23:21.573427 22234 consensus_queue.cc:260] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:23:21.574015 22234 ts_tablet_manager.cc:1398] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 08:23:21.576537 28655 tablet_copy_service.cc:342] P 4574396f35514c9caccb346f1a7af84d: Request end of tablet copy session 716ffbb2a00b4f43b03e5cedcc9d302a-28e42c3244f949c7b7343bf587894ac5 received from {username='slave'} at 127.20.107.130:41829
I20240503 08:23:21.576690 28655 tablet_copy_service.cc:433] P 4574396f35514c9caccb346f1a7af84d: ending tablet copy session 716ffbb2a00b4f43b03e5cedcc9d302a-28e42c3244f949c7b7343bf587894ac5 on tablet 28e42c3244f949c7b7343bf587894ac5 with peer 716ffbb2a00b4f43b03e5cedcc9d302a
W20240503 08:23:21.583561 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 108 similar messages]
I20240503 08:23:21.587846 22239 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:23:21.588188 22239 tablet_bootstrap.cc:492] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: Bootstrap complete.
I20240503 08:23:21.588410 22239 ts_tablet_manager.cc:1367] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.001s
I20240503 08:23:21.588994 22239 raft_consensus.cc:348] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:23:21.589169 22239 raft_consensus.cc:729] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 716ffbb2a00b4f43b03e5cedcc9d302a, State: Initialized, Role: LEARNER
I20240503 08:23:21.589336 22239 consensus_queue.cc:260] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: NON_VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: true } }
I20240503 08:23:21.589769 22239 ts_tablet_manager.cc:1398] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:23:21.590499 28785 tablet_copy_service.cc:342] P 93f7e733cec9415eb422b0a7ba8621cb: Request end of tablet copy session 716ffbb2a00b4f43b03e5cedcc9d302a-4babf0ea9b954a23b1b1c4e403dce0bd received from {username='slave'} at 127.20.107.130:35911
I20240503 08:23:21.590597 28785 tablet_copy_service.cc:433] P 93f7e733cec9415eb422b0a7ba8621cb: ending tablet copy session 716ffbb2a00b4f43b03e5cedcc9d302a-4babf0ea9b954a23b1b1c4e403dce0bd on tablet 4babf0ea9b954a23b1b1c4e403dce0bd with peer 716ffbb2a00b4f43b03e5cedcc9d302a
I20240503 08:23:21.610023 28635 raft_consensus.cc:1203] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
W20240503 08:23:21.655928 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 08:23:21.699370 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240503 08:23:21.714438 21965 raft_consensus.cc:1050] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER 93f7e733cec9415eb422b0a7ba8621cb to VOTER
I20240503 08:23:21.715262 21965 consensus_queue.cc:237] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 5, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } }
I20240503 08:23:21.717391 28502 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 3 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:23:21.717438 28762 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb [term 3 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:23:21.717958 28374 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 3 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:23:21.718469 21972 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:23:21.718758 22045 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:23:21.719318 21972 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:23:21.723378 22045 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:21.724077 28374 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:21.728979 28502 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:21.729447 28762 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:21.729939 28226 catalog_manager.cc:5484] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 10 to 11, 93f7e733cec9415eb422b0a7ba8621cb (127.20.107.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:21.736164 28635 consensus_queue.cc:237] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } }
I20240503 08:23:21.738791 28502 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 3 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 08:23:21.739396 21965 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:23:21.741884 21965 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:21.743927 28761 raft_consensus.cc:1261] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb [term 3 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 08:23:21.744621 28502 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:21.745383 22045 consensus_queue.cc:1035] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:23:21.746255 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba833ce23c4c4e81bc1a81125a1208c9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:21.747068 28761 raft_consensus.cc:2932] T ba833ce23c4c4e81bc1a81125a1208c9 P 93f7e733cec9415eb422b0a7ba8621cb [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } } }
I20240503 08:23:21.750162 28226 catalog_manager.cc:5484] T ba833ce23c4c4e81bc1a81125a1208c9 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 11 to 12, VOTER 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129) evicted. New cstate: current_term: 3 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:23:21.755422 28353 tablet_service.cc:1514] Processing DeleteTablet for tablet ba833ce23c4c4e81bc1a81125a1208c9 with delete_type TABLET_DATA_TOMBSTONED (TS 4e4a1242f3714d03a6c9ee51dd63929c not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:36186
I20240503 08:23:21.756245 22261 tablet_replica.cc:330] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c: stopping tablet replica
I20240503 08:23:21.756440 22261 raft_consensus.cc:2229] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:23:21.756852 22261 raft_consensus.cc:2258] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:23:21.758137 22261 ts_tablet_manager.cc:1875] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:23:21.770881 22261 ts_tablet_manager.cc:1888] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240503 08:23:21.771154 22261 log.cc:1196] T ba833ce23c4c4e81bc1a81125a1208c9 P 4e4a1242f3714d03a6c9ee51dd63929c: Deleting WAL directory at /tmp/dist-test-taskZB6CEZ/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723866735031-20910-0/raft_consensus-itest-cluster/ts-0/wal/wals/ba833ce23c4c4e81bc1a81125a1208c9
I20240503 08:23:21.771997 28218 catalog_manager.cc:4830] TS 4e4a1242f3714d03a6c9ee51dd63929c (127.20.107.129:38529): tablet ba833ce23c4c4e81bc1a81125a1208c9 (table rebalance_test_table_8 [id=649380be96b24c78aa15b40471703d93]) successfully deleted
W20240503 08:23:21.772681 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 08:23:21.807610 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(4babf0ea9b954a23b1b1c4e403dce0bd): 4923 bytes on disk
I20240503 08:23:21.808137 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(4babf0ea9b954a23b1b1c4e403dce0bd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 08:23:21.809150 28551 maintenance_manager.cc:419] P 716ffbb2a00b4f43b03e5cedcc9d302a: Scheduling UndoDeltaBlockGCOp(28e42c3244f949c7b7343bf587894ac5): 4583 bytes on disk
I20240503 08:23:21.809564 28442 maintenance_manager.cc:643] P 716ffbb2a00b4f43b03e5cedcc9d302a: UndoDeltaBlockGCOp(28e42c3244f949c7b7343bf587894ac5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240503 08:23:21.889099 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 08:23:21.902863 28635 raft_consensus.cc:1203] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
W20240503 08:23:21.937423 28217 catalog_manager.cc:4573] TS c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 564fb1b51abd4f8da0372ec99e3c599c: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
I20240503 08:23:22.027925 28502 raft_consensus.cc:1203] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240503 08:23:22.106065 22214 raft_consensus.cc:1050] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER 4574396f35514c9caccb346f1a7af84d to VOTER
I20240503 08:23:22.106737 22214 consensus_queue.cc:237] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } }
I20240503 08:23:22.108963 28635 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:23:22.109163 28502 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:23:22.109536 22293 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:23:22.109936 22214 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 08:23:22.110116 28308 consensus_peers.cc:485] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:22.118664 22293 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:22.119182 28502 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:22.122053 28635 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:22.123364 28232 catalog_manager.cc:5484] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 12 to 13, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:23:22.128939 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 08:23:22.128960 28374 consensus_queue.cc:237] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } }
I20240503 08:23:22.130535 28635 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:23:22.130854 28502 raft_consensus.cc:1261] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:23:22.131070 22214 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:23:22.132282 22214 consensus_queue.cc:1035] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:23:22.133143 22293 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:22.133773 28635 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:22.135666 28501 raft_consensus.cc:2932] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:22.138844 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e9997e8da99c4f45a63bc3db521b33ba with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:22.140477 28502 raft_consensus.cc:1203] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:23:22.140611 28226 catalog_manager.cc:5484] T e9997e8da99c4f45a63bc3db521b33ba P 716ffbb2a00b4f43b03e5cedcc9d302a reported cstate change: config changed from index 13 to 14, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
W20240503 08:23:22.185847 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:23:22.239887 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:23:22.247210 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:23:22.250031 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:23:22.482076 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:23:22.505479 22293 raft_consensus.cc:1050] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c: attempting to promote NON_VOTER 4574396f35514c9caccb346f1a7af84d to VOTER
I20240503 08:23:22.506157 22293 consensus_queue.cc:237] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } }
I20240503 08:23:22.508091 28635 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 LEARNER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:23:22.508181 28502 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:23:22.508611 22152 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:23:22.509065 22214 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 08:23:22.509222 28308 consensus_peers.cc:485] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:22.511457 22214 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:22.512023 28635 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:22.516326 28502 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:22.518028 28232 catalog_manager.cc:5484] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 13 to 14, 4574396f35514c9caccb346f1a7af84d (127.20.107.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:22.523029 28374 consensus_queue.cc:237] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } }
W20240503 08:23:22.523922 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:23:22.524631 28635 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240503 08:23:22.524670 28502 raft_consensus.cc:1261] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4e4a1242f3714d03a6c9ee51dd63929c: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240503 08:23:22.525104 22218 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:23:22.525502 22218 consensus_queue.cc:1035] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:23:22.527177 22218 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:22.527747 28635 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:22.528412 28501 raft_consensus.cc:2932] T 48e31346107a453a81921b23b244e7ff P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } }
I20240503 08:23:22.530223 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48e31346107a453a81921b23b244e7ff with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:22.532816 28226 catalog_manager.cc:5484] T 48e31346107a453a81921b23b244e7ff P 4e4a1242f3714d03a6c9ee51dd63929c reported cstate change: config changed from index 14 to 15, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New cstate: current_term: 1 leader_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:23:22.561734 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240503 08:23:22.617699 22291 raft_consensus.cc:1050] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb: attempting to promote NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a to VOTER
I20240503 08:23:22.618393 22291 consensus_queue.cc:237] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } }
W20240503 08:23:22.619124 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240503 08:23:22.620311 28501 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:23:22.620911 28635 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:23:22.621210 22321 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:23:22.621585 22323 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 08:23:22.621781 28699 consensus_peers.cc:485] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:22.626125 22291 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:22.626718 28635 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:22.629938 28501 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:22.631103 28232 catalog_manager.cc:5484] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 12 to 13, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:22.635757 28761 consensus_queue.cc:237] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } }
I20240503 08:23:22.637477 28501 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:23:22.637934 22324 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:23:22.639689 22323 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:22.641270 28635 raft_consensus.cc:1261] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Refusing update from remote peer 93f7e733cec9415eb422b0a7ba8621cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:23:22.641724 22324 consensus_queue.cc:1035] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:23:22.642702 28218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4babf0ea9b954a23b1b1c4e403dce0bd with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:22.643039 28501 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:22.646991 28635 raft_consensus.cc:2932] T 4babf0ea9b954a23b1b1c4e403dce0bd P 4574396f35514c9caccb346f1a7af84d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:22.653309 28226 catalog_manager.cc:5484] T 4babf0ea9b954a23b1b1c4e403dce0bd P 93f7e733cec9415eb422b0a7ba8621cb reported cstate change: config changed from index 13 to 14, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New cstate: current_term: 1 leader_uuid: "93f7e733cec9415eb422b0a7ba8621cb" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:22.691105 21965 raft_consensus.cc:1050] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d: attempting to promote NON_VOTER 716ffbb2a00b4f43b03e5cedcc9d302a to VOTER
I20240503 08:23:22.691955 21965 consensus_queue.cc:237] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } }
I20240503 08:23:22.694447 28501 raft_consensus.cc:1261] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 LEARNER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:23:22.694864 28374 raft_consensus.cc:1261] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:23:22.695344 28569 consensus_peers.cc:485] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:23:22.695434 22338 consensus_queue.cc:1035] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:23:22.695772 21965 consensus_queue.cc:1035] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:23:22.698185 22338 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:22.698663 28501 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:22.702661 28374 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:22.706058 28232 catalog_manager.cc:5484] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 7 to 8, 716ffbb2a00b4f43b03e5cedcc9d302a (127.20.107.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:23:22.710585 28635 consensus_queue.cc:237] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } }
I20240503 08:23:22.712080 28501 raft_consensus.cc:1261] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:23:22.712159 28374 raft_consensus.cc:1261] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Refusing update from remote peer 4574396f35514c9caccb346f1a7af84d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:23:22.712538 22338 consensus_queue.cc:1035] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:22.712978 22045 consensus_queue.cc:1035] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:23:22.715020 22045 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:22.715596 28501 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 716ffbb2a00b4f43b03e5cedcc9d302a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:22.715827 28374 raft_consensus.cc:2932] T 28e42c3244f949c7b7343bf587894ac5 P 4e4a1242f3714d03a6c9ee51dd63929c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } } }
I20240503 08:23:22.718024 28217 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28e42c3244f949c7b7343bf587894ac5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:23:22.719501 28226 catalog_manager.cc:5484] T 28e42c3244f949c7b7343bf587894ac5 P 4574396f35514c9caccb346f1a7af84d reported cstate change: config changed from index 8 to 9, VOTER 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134) evicted. New cstate: current_term: 1 leader_uuid: "4574396f35514c9caccb346f1a7af84d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4e4a1242f3714d03a6c9ee51dd63929c" member_type: VOTER last_known_addr { host: "127.20.107.129" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4574396f35514c9caccb346f1a7af84d" member_type: VOTER last_known_addr { host: "127.20.107.131" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716ffbb2a00b4f43b03e5cedcc9d302a" member_type: VOTER last_known_addr { host: "127.20.107.130" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:23:22.723438 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:23:22.767462 28218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240503 08:23:22.787573 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:23:22.861588 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:23:22.956920 28217 catalog_manager.cc:4573] TS c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0caf86850b3841ea811b0a6ab19ae330: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:23.092386 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:23:23.264801 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:23:23.349632 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:23:23.571406 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 08:23:23.597612 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:23:23.633471 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:23:23.763104 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 08:23:23.794966 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:23:23.865363 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 08:23:23.905349 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:23:23.932209 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:23:24.005209 28217 catalog_manager.cc:4573] TS c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 564fb1b51abd4f8da0372ec99e3c599c: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:24.168156 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:24.171167 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:23:24.244760 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:23:24.446048 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:23:24.453408 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:23:24.518882 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:23:24.625036 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 08:23:24.768517 28314 tablet_copy_service.cc:454] P 4e4a1242f3714d03a6c9ee51dd63929c: tablet copy session bea2145b8e24436392b18ec3cfe90b93-173376257c4946b5a801a27e4dffe90d on tablet 173376257c4946b5a801a27e4dffe90d with peer bea2145b8e24436392b18ec3cfe90b93 has expired after 607.228s of idle time
I20240503 08:23:24.768693 28314 tablet_copy_service.cc:433] P 4e4a1242f3714d03a6c9ee51dd63929c: ending tablet copy session bea2145b8e24436392b18ec3cfe90b93-173376257c4946b5a801a27e4dffe90d on tablet 173376257c4946b5a801a27e4dffe90d with peer bea2145b8e24436392b18ec3cfe90b93
W20240503 08:23:24.832851 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:23:24.839498 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:23:24.934263 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:23:25.045493 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:23:25.101329 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:23:25.263914 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:23:25.281198 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:23:25.293661 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240503 08:23:25.295861 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 08:23:25.346249 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:23:25.525519 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240503 08:23:25.541707 22362 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:23:25.541863 22362 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:23:25.542395 22362 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:23:25.543763 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:25.543903 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:23:25.543999 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:23:25.544202 22362 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:23:25.569083 28219 catalog_manager.cc:4573] TS bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b5736ee8065344ea8124fdf759035216: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111)
W20240503 08:23:25.665825 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:23:25.725996 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:23:25.933272 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:23:25.962831 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:23:26.002250 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:23:26.020759 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:23:26.066114 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 08:23:26.113818 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:23:26.325069 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:23:26.371795 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:23:26.445107 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:23:26.483198 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:23:26.666208 28568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240503 08:23:26.683171 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:26.706801 28217 catalog_manager.cc:4573] TS c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ede24a27608a4cf196dd46f85c4163e1: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:26.755885 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:23:26.785650 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240503 08:23:26.826467 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:23:26.828346 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:23:26.882560 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:23:27.159171 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:23:27.169847 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:27.247671 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:23:27.385759 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:27.563443 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:23:27.691965 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:23:27.710124 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:23:27.817605 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:27.861605 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:27.906368 28217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:23:27.907436 28217 catalog_manager.cc:4573] TS c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6e9537f30aaf45adbc2cc4a784009de7: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111)
W20240503 08:23:27.961750 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:28.021114 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:28.096272 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:23:28.163887 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:23:28.384388 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:23:28.444255 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:23:28.537364 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:23:28.624101 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:23:28.639106 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:28.703188 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:28.748135 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:23:28.764281 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:23:28.903421 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:23:29.160133 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:29.309053 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:29.356789 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:23:29.465994 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:29.484479 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:29.622308 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:23:29.668736 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:30.095544 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:23:30.098909 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:30.107817 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:23:30.203821 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:23:30.262828 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:23:30.347999 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:30.420399 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 08:23:30.440934 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:30.489338 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:30.551195 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:23:30.600665 28218 catalog_manager.cc:4573] TS 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e9997e8da99c4f45a63bc3db521b33ba: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:30.601389 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:30.705222 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:30.840234 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:31.009689 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:23:31.083972 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:23:31.086555 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:23:31.203168 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:31.276464 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:31.280094 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:23:31.315078 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:23:31.363229 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:23:31.515460 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:31.617225 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:31.881389 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240503 08:23:31.909664 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:31.924782 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:31.937237 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:23:31.984911 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:32.030711 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:23:32.031524 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:23:32.192798 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:32.637307 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:23:32.640435 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:32.742780 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:32.757117 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:23:32.993598 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:33.008841 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:33.090780 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:23:33.092351 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:33.256626 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:33.296007 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:33.369323 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:23:33.423164 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:23:33.494892 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:23:33.654588 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:33.693152 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:23:33.722594 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:33.772569 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:33.850591 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:23:34.085297 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:34.149705 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:23:34.196552 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:34.207515 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:34.345475 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:34.348613 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:34.348855 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:23:34.429826 28218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240503 08:23:34.430931 28218 catalog_manager.cc:4573] TS b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 173376257c4946b5a801a27e4dffe90d: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:34.438287 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:34.624532 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:23:34.946269 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:23:35.158524 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:35.227409 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:23:35.345974 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:35.405992 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:35.431210 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:35.436611 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240503 08:23:35.638132 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:35.676573 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:35.727098 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:35.765446 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:23:35.811517 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:23:35.819481 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:35.869797 28218 catalog_manager.cc:4573] TS b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 09156806b0744752969b6a574e24b50a: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:35.924093 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:23:35.961701 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:23:36.023805 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:36.096108 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:23:36.332494 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:36.337733 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:23:36.424037 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:36.434185 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:23:36.650472 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:36.690591 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:36.801600 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:36.815915 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:36.829051 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:23:36.832142 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:36.909987 28218 catalog_manager.cc:4573] TS b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ba833ce23c4c4e81bc1a81125a1208c9: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:36.912902 28568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240503 08:23:36.913807 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:23:37.175269 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:23:37.369436 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:23:37.407991 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:37.593595 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:23:37.667672 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:37.678491 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:23:37.847766 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:37.981776 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:38.177819 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:23:38.178756 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:38.278913 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:23:38.292409 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:38.308807 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:23:38.328321 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:23:38.338142 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:38.366235 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:38.393275 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:38.694372 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240503 08:23:38.696702 23054 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:23:38.697036 23054 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:23:38.697729 23054 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:23:38.699084 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:38.699224 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:23:38.699301 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:23:38.699517 23120 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:23:38.808513 28218 catalog_manager.cc:4573] TS 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e9997e8da99c4f45a63bc3db521b33ba: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:38.854985 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:23:38.939662 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:38.998157 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:39.086548 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:23:39.276077 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:39.331713 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:23:39.450186 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:39.472438 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:39.481344 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:39.664745 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:23:39.842177 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:23:39.876267 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:40.104250 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:40.145982 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:23:40.245178 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:23:40.439023 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 08:23:40.439872 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:40.469020 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:40.610366 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:40.696725 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:40.731127 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:40.815202 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:40.855270 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:23:40.859798 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:23:40.869436 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:40.934422 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:23:41.132316 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:23:41.133147 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:41.170135 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:23:41.215174 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:41.334343 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:23:41.409458 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:23:41.637104 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:23:41.731906 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:41.876349 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:41.904525 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:41.924728 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240503 08:23:42.067880 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:42.091109 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:42.182523 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:23:42.286938 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:23:42.373378 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:42.444620 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:23:42.450868 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:42.617689 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:23:42.867278 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:23:42.981739 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:43.112526 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:43.208559 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:43.265410 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:43.368132 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:43.372088 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:43.385453 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:43.389375 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:23:43.452488 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:23:43.589337 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:23:43.611300 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:23:43.671206 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:43.722620 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:23:43.812037 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:43.841617 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:23:44.094120 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:44.170661 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:23:44.361486 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:44.401124 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:44.610160 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:23:44.678333 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:44.713207 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:23:44.727025 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:44.909456 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:44.968597 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:45.024296 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:23:45.332343 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:23:45.379429 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:45.531123 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240503 08:23:45.531925 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:45.534683 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:45.591861 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:23:45.650455 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:45.770447 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:45.884742 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:45.964229 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:23:46.050534 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:23:46.054457 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:46.142674 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:46.182060 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:23:46.182873 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:23:46.210853 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:46.215662 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:23:46.422935 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:23:46.486495 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:23:46.617051 23424 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:23:46.617233 23424 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:23:46.617795 23424 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:23:46.619257 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:46.619405 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:23:46.619482 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:23:46.619704 23424 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:23:46.626027 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:46.868036 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:46.929124 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 08:23:46.968508 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:47.017398 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:47.131601 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:23:47.152529 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:47.228327 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:23:47.310532 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:23:47.339712 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:47.566951 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:47.837910 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:47.846125 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:23:47.874758 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:23:47.926973 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:47.977619 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:48.099056 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:48.109786 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:23:48.272826 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:23:48.363477 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:48.408747 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:23:48.452188 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:48.590018 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:48.692667 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:23:48.713112 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:48.723939 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:23:48.747596 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:48.987705 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:23:49.057054 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:49.073802 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:23:49.310695 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:49.363909 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:49.479449 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:49.577111 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:49.620265 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:49.690153 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:23:49.740723 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:23:50.059934 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:50.147790 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:23:50.371752 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:23:50.397856 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:50.404692 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:50.481668 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:50.572472 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240503 08:23:50.673421 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:50.673619 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:23:50.676048 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:23:50.758807 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:50.781152 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:50.856499 28218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 08:23:50.857620 28218 catalog_manager.cc:4573] TS b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 173376257c4946b5a801a27e4dffe90d: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:50.891808 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:23:50.953610 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:23:51.047147 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:23:51.080253 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:51.130043 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:51.179587 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:51.183373 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:23:51.590843 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:51.601408 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:51.618615 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:23:51.790396 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:23:51.807507 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:52.008744 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:52.020747 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240503 08:23:52.124385 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:23:52.240489 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:52.290418 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:23:52.292737 28218 catalog_manager.cc:4573] TS b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 09156806b0744752969b6a574e24b50a: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:52.385396 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:23:52.525997 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:23:52.546505 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:52.549232 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:23:52.782500 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:52.846336 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:23:52.890515 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:52.937271 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:52.986315 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:23:52.996718 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:53.208174 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:53.249043 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:53.272408 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:23:53.300796 28218 catalog_manager.cc:4573] TS b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ba833ce23c4c4e81bc1a81125a1208c9: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:23:53.350574 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:23:53.356971 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:53.432504 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:23:53.742085 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:23:53.773005 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:53.917376 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:23:53.961270 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:53.998558 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:54.331574 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:23:54.355448 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:23:54.448697 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:54.695335 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:23:54.705152 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:54.715606 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:23:54.830055 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:23:54.907770 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:23:55.069926 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:55.136123 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:23:55.205235 28218 catalog_manager.cc:4573] TS 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e9997e8da99c4f45a63bc3db521b33ba: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
W20240503 08:23:55.215386 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:23:55.236039 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:23:55.356532 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:23:55.387115 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:23:55.409936 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:55.557725 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:23:55.696826 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240503 08:23:55.697685 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:55.878094 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:23:55.919729 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:23:55.953460 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:23:56.104383 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:23:56.117873 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:23:56.141366 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:23:56.191677 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 08:23:56.202562 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:23:56.289146 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:56.628767 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:23:56.646270 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:23:56.653244 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:23:56.663990 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:23:57.092303 28570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 08:23:57.093115 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:23:57.139854 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:23:57.211697 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:23:57.366353 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:23:57.392339 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:23:57.393263 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:23:57.461906 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:23:57.599792 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:23:57.611712 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:23:57.862128 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:23:57.946699 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:23:57.988847 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:23:57.988847 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:23:57.994743 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:23:58.009922 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:23:58.068231 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:23:58.229647 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:58.257189 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:23:58.477663 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:23:58.494252 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:23:58.715137 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:23:58.795531 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:23:58.824558 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:23:58.831954 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:23:58.860232 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:23:59.076974 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:23:59.169620 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:23:59.177867 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:23:59.563283 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:23:59.746320 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:23:59.774132 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:23:59.792192 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:23:59.872107 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:23:59.939304 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:23:59.954643 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:24:00.027412 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:24:00.210110 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:24:00.340533 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:24:00.414572 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:24:00.425668 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:24:00.571884 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:24:00.594100 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:24:00.607761 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:24:00.704226 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 08:24:00.746064 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:24:00.751435 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:24:00.821936 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:24:01.073532 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:24:01.158823 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:24:01.299880 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:24:01.300716 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:24:01.334375 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:01.338596 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:24:01.352823 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:24:01.538172 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:24:01.786710 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:24:01.841991 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:24:01.877287 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:24:02.127912 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 08:24:02.166363 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:24:02.192781 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:24:02.199013 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:24:02.312350 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:02.358134 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:02.382037 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:02.412520 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:24:02.453912 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:24:02.640807 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:02.834825 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:02.974411 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:03.030385 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:24:03.102523 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:03.103384 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:24:03.111694 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:24:03.122834 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:24:03.174342 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:24:03.304822 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:24:03.427453 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:24:03.674130 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:03.846359 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:24:03.855829 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:24:03.909569 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:03.955325 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:24:04.194227 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:24:04.200520 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:04.270412 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:04.378537 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:24:04.627131 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:04.640177 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:24:04.661243 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:24:04.698817 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:04.706414 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:04.869076 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:05.009963 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:05.195402 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:05.283113 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:05.532368 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:05.554051 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:05.602320 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:05.634167 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:24:05.687216 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:24:05.761706 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:24:05.763712 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 08:24:05.764603 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:24:05.841472 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:24:05.900236 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:24:06.011621 24082 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:24:06.011801 24082 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:24:06.012442 24082 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:24:06.013852 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:24:06.014032 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:24:06.014118 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:24:06.014325 24082 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:24:06.297813 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:06.396893 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:24:06.410470 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:06.533603 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:24:06.559589 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:24:06.611956 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:06.646595 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:24:06.657574 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:06.752851 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:06.826467 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:24:06.973505 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:24:07.073159 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:24:07.107342 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:07.201552 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:07.227346 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 08:24:07.258322 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:07.541234 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:07.554658 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 08:24:07.555326 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:07.744277 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:07.761696 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:07.766719 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:07.973809 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:08.016849 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:24:08.045153 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:24:08.140174 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:08.147773 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:08.176108 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:24:08.218597 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:24:08.566711 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:24:08.746690 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:08.790750 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:24:08.852762 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:08.922681 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:09.099328 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:24:09.125258 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:09.204667 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:24:09.230129 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:09.253804 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:24:09.364035 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:24:09.589190 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:09.629354 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:09.764948 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:24:09.795912 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:09.956907 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:10.008620 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:10.160708 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:10.252772 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:10.382691 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:10.448774 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:10.554317 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:24:10.611054 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:24:10.615072 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:10.738260 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:10.787240 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:24:10.840932 28439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240503 08:24:11.044498 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:24:11.099138 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:24:11.164446 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:11.425426 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:11.472409 28308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:24:11.479012 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:11.481679 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:11.501714 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:24:11.693745 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:24:11.699021 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:24:11.773563 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:24:11.900117 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:11.995191 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:24:12.056008 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:12.078569 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:24:12.126848 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:12.250411 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240503 08:24:12.438668 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:12.504328 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:12.576606 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:24:12.577415 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:12.579452 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:12.587882 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:24:12.727389 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:13.009500 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:13.023227 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:24:13.057456 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:13.090054 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:24:13.120476 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:13.243405 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:24:13.416092 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:13.481837 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:24:13.560134 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:24:13.693486 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:24:13.889554 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:24:14.026022 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:14.082860 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:24:14.164727 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:14.215652 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:24:14.382071 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:24:14.458915 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:14.505259 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:24:14.555608 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:24:14.694212 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:14.784796 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:24:14.801633 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:14.883137 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:24:15.019626 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:24:15.159380 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:24:15.207767 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:24:15.388283 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:24:15.457938 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:15.495302 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:24:15.503749 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:24:15.541261 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:15.644470 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:24:15.655079 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:24:15.742516 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:15.835000 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:24:15.865377 28438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 08:24:16.176684 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:24:16.205816 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:24:16.269593 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:24:16.323930 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:24:16.413548 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:24:16.510481 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:24:16.780727 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:24:16.865125 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:24:16.865105 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:24:17.034597 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:24:17.063053 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:24:17.165764 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:24:17.242100 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240503 08:24:17.257356 28568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 08:24:17.345749 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:24:17.353962 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:24:17.430740 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:24:17.617877 28699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:24:17.668371 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:24:17.722057 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:24:17.791561 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:24:17.897732 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:24:18.041347 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:24:18.073495 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:24:18.087841 28438 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:24:18.096441 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:24:18.112649 28308 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:24:18.210541 28699 consensus_peers.cc:485] T 0caf86850b3841ea811b0a6ab19ae330 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:24:18.244853 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:24:18.387462 28569 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240503 08:24:18.456619 24806 raft_consensus.cc:482] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 21d21a5b596b4d48a17a1fb54a972844)
I20240503 08:24:18.456832 24806 raft_consensus.cc:504] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "93f7e733cec9415eb422b0a7ba8621cb" member_type: VOTER last_known_addr { host: "127.20.107.132" port: 34095 } } peers { permanent_uuid: "b29361eb4afa4ab3af7d8019d8a5042d" member_type: VOTER last_known_addr { host: "127.20.107.136" port: 42389 } attrs { promote: false } } peers { permanent_uuid: "21d21a5b596b4d48a17a1fb54a972844" member_type: VOTER last_known_addr { host: "127.20.107.134" port: 34587 } attrs { promote: false } }
I20240503 08:24:18.457589 24806 leader_election.cc:288] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389), 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587)
W20240503 08:24:18.458837 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111)
W20240503 08:24:18.459002 28699 leader_election.cc:334] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111)
I20240503 08:24:18.459065 28699 leader_election.cc:302] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f7e733cec9415eb422b0a7ba8621cb; no voters: 21d21a5b596b4d48a17a1fb54a972844, b29361eb4afa4ab3af7d8019d8a5042d
I20240503 08:24:18.459241 24806 raft_consensus.cc:2726] T 9b3b8b80ebd44f27899f97211266c7b2 P 93f7e733cec9415eb422b0a7ba8621cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:24:18.537765 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:24:18.760502 28309 consensus_peers.cc:485] T ede24a27608a4cf196dd46f85c4163e1 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:24:18.881377 28308 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:24:18.887418 28569 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240503 08:24:18.939033 28438 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:24:19.243182 28307 consensus_peers.cc:485] T d44335d8be7245b1bb52b2eab9e42c18 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:24:19.414994 28568 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:24:19.588950 28308 consensus_peers.cc:485] T 173376257c4946b5a801a27e4dffe90d P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:24:19.666280 28308 consensus_peers.cc:485] T d13c33b8712e4c488abb99b73ff42b86 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:24:19.666311 28568 consensus_peers.cc:485] T 585ba864cc384e7f9db90e9d4c47e82c P 4574396f35514c9caccb346f1a7af84d -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240503 08:24:19.759799 28439 consensus_peers.cc:485] T b0d07a6278d545e69bf7d2012109d1f4 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:24:19.801208 28570 consensus_peers.cc:485] T b4ceeeb3495e4a8fac6e13e431c64af4 P 4574396f35514c9caccb346f1a7af84d -> Peer bea2145b8e24436392b18ec3cfe90b93 (127.20.107.135:45937): Couldn't send request to peer bea2145b8e24436392b18ec3cfe90b93. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.135:45937: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:24:19.808363 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240503 08:24:19.850093 28699 consensus_peers.cc:485] T c6e1b9f104ea471cab7527bd56433327 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:24:20.011765 28307 consensus_peers.cc:485] T b2ebb0b8475b4ed9a09711a87867de08 P 4e4a1242f3714d03a6c9ee51dd63929c -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240503 08:24:20.240618 28438 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer c8375219addc48fa99ac474ed1c9aef9 (127.20.107.133:35121): Couldn't send request to peer c8375219addc48fa99ac474ed1c9aef9. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.133:35121: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:24:20.261543 28699 consensus_peers.cc:485] T 3bfa8537bc8a4950b53fbf3c36d5ecf3 P 93f7e733cec9415eb422b0a7ba8621cb -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240503 08:24:20.311930 28569 consensus_peers.cc:485] T d60a162155504975a39c20eff88341e1 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:24:20.339377 28569 consensus_peers.cc:485] T 2d113e8de02f447b9da0a1cb03686cd2 P 4574396f35514c9caccb346f1a7af84d -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240503 08:24:20.340173 28439 consensus_peers.cc:485] T ac392ff7965a405f96efc777aca4f0bb P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer b29361eb4afa4ab3af7d8019d8a5042d (127.20.107.136:42389): Couldn't send request to peer b29361eb4afa4ab3af7d8019d8a5042d. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.136:42389: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240503 08:24:20.543628 28439 consensus_peers.cc:485] T a20489bf65774902af38d228ee966ffe P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn't send request to peer 21d21a5b596b4d48a17a1fb54a972844. Status: Network error: Client connection negotiation failed: client connection to 127.20.107.134:34587: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:24:20.610906 28439 consensus_peers.cc:485] T b5736ee8065344ea8124fdf759035216 P 716ffbb2a00b4f43b03e5cedcc9d302a -> Peer 21d21a5b596b4d48a17a1fb54a972844 (127.20.107.134:34587): Couldn'